Millimeter wave sensor used to optimize performance of a beamforming microphone array

ABSTRACT

A method for operating a beamforming microphone array for use in a predetermined area is provided herein, the method comprising: receiving acoustic audio signals at each of a plurality of microphones, converting the same to an electrical mic audio signal, and outputting each of the plurality of electrical mic audio signals; generating a user location data signal by a wave sensor system, and outputting the user location data signal, wherein the user location data signal includes location information of one or more people within the predetermined area; receiving both the user location data signal and plurality of echo-corrected mic audio signals at an adaptive beamforming device; and adapting one or more beams by the adaptive beamforming device based on the user location data signal and plurality of mic audio signals wherein each of the one or more beams acquires sound from one or more specific locations in the predetermined area.

PRIORITY INFORMATION

The present application claims priority under 35 U.S.C. § 119(e) to U.S.Provisional Patent Application Ser. No. 62/811,007, filed on Feb. 27,2019, the entire contents of which are expressly incorporated herein byreference.

CROSS REFERENCE TO RELATED APPLICATIONS

Related subject matter is disclosed in the following U.S.Non-provisional patent applications: Client Matter No. CP00503-01, Ser.No. 16/801,964, filed Feb. 26, 2020; Client Matter No. CP00503-02, Ser.No. 16/802,004, filed Feb. 26, 2020; and Client Matter No. CP00503-04,Ser. No. 16/802,111, filed Feb. 26, 2020, the entire contents of all ofwhich are expressly incorporated herein by reference.

BACKGROUND Technical Field

Aspects of the embodiments relate to audio systems, and morespecifically to systems, methods, and modes for implementing amillimeter wave sensor to optimize operation of a beamforming microphonearray, as well as for use in other home or enterprise systems.

Background Art

Microphone (mic) arrays can be used in currently availableaudio-conferencing systems instead of a single mic located on aconference room table. Such mic arrays typically have two or more mics,and they also typically employ beamforming techniques to increase theirability to pick up and isolate the voices of the people participating inthe audio conference. There are primarily two types of ceiling micarrays currently being used for audio (and video) conferencing systems,each of which exhibits significant drawbacks. The first type is referredto as a fixed beam type, wherein during installation beams are manuallypositioned over the locations that people will likely sit. A computer isrequired for this setup. This type of mic array can have multipleoutputs (one for each beam) or a single output from a built-in mixer.The beams have to be configured to be large enough to cover spaces wherepeople are likely to be located during an audio conference. Large beamcoverage positions, however, have lower S/N performance, especially forpositions that are located at a significant distance from the mic array.If one or more people move during the conference call and move in andout of “their” beam position, dropouts in the audio can occur—meaningtheir voices are less likely to be heard or become less clear.

The other type of mic array is a dynamic beamformer. The dynamicbeamformer type of ceiling mic array uses one or more algorithms tolocate the position of someone talking and adapts the beam to thatlocation. However, such systems are susceptible to “false positives,”meaning that a dynamic beamformer cannot distinguish between aconversation meant for the audio conference that is occurring, and theconversation that might be happening in the hallway just outside thedoor to the conference room. Also, other sources of noise can cause thedynamic beams to focus on them. Such sources of noise can include thespeakers that re-produce the far end audio ringers on cellular devices,fan noises, air conditioning or heating noises, among others.Sophisticated and therefore expensive software and/or additional manualsetup can diminish but not eliminate such problems.

Thus, currently available beamforming microphone arrays have limitationssuch as non-optimized beam forming parameters. Beam position and area ofcoverage will be non-optimal. Adaptive beamformers cannot distinguishvoice from people vs. audio speakers. As those of skill in the art cantherefore appreciate, both of the primary currently available devicestherefore exhibit significant drawbacks.

Accordingly, a need has arisen for systems, methods, and modes forsystems, methods, and modes for implementing a millimeter wave sensor tooptimize operation of a beamforming microphone array.

SUMMARY

It is an object of the embodiments to substantially solve at least theproblems and/or disadvantages discussed above, and to provide at leastone or more of the advantages described below.

It is therefore a general aspect of the embodiments to provide systems,methods, and modes for systems, methods, and modes for implementing amillimeter wave sensor to optimize operation of a beamforming microphonearray that will obviate or minimize problems of the type previouslydescribed.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Further features and advantages of the aspects of the embodiments, aswell as the structure and operation of the various embodiments, aredescribed in detail below with reference to the accompanying drawings.It is noted that the aspects of the embodiments are not limited to thespecific embodiments described herein. Such embodiments are presentedherein for illustrative purposes only. Additional embodiments will beapparent to persons skilled in the relevant art(s) based on theteachings contained herein.

According to a first aspect of the embodiments, a beamforming microphonearray is provided, comprising: a plurality of microphones each of whichis adapted to receive an acoustic audio signal and convert the same to amicrophone (mic) audio signal; a wave sensor system adapted to determinelocations of one or more people within a predetermined area about thebeamforming microphone array and output the same as user location datasignal; and an adaptive beamforming circuit adapted to receive the userlocation data signal and plurality of mic audio signals and performadaptive beamforming on the plurality of mic audio signals that takesinto account the received user location data signal to adapt one or morebeams to acquire sound from one or more specific locations in thepredetermined area.

According to the first aspect of the embodiments, the wave sensor systemcomprises: a millimeter (mm) wave transmitter; and a wave receiver.

According to the first aspect of the embodiments, the wave sensor systemcomprises: an optical transmitter; and an optical receiver.

According to the first aspect of the embodiments, the wave sensor systemis further adapted to generate a three dimensional image of thepredetermined area and output the same as an area image data signal.

According to the first aspect of the embodiments, the adaptivebeamforming circuit is further adapted to receive the area image datasignal and the plurality of mic audio signals and perform adaptivebeamforming on the plurality of mic audio signals that takes intoaccount the received area image data signal to adapt one or more beamsto acquire sound from one or more specific locations in thepredetermined area.

According to the first aspect of the embodiments, the adaptivebeamforming circuit is adapted to modify the beam audio signals toreduce noise reflected off one or more objects within the predeterminedarea based on the area image data signal.

According to the first aspect of the embodiments, the area image datasignal comprises: information as to where motion is occurring within thepredetermined area.

According to the first aspect of the embodiments, the informationcontained within the area image data signal that motion is occurringwithin the predetermined area substantially eliminates objects that aresubstantially at rest.

According to the first aspect of the embodiments, the informationcontained within the area image data signal that motion is occurringwithin the predetermined area does not include objects that move with asubstantial constant velocity.

According to the first aspect of the embodiments, the object that moveswith a substantially constant periodicity comprises a fan.

According to the first aspect of the embodiments, the area image datasignal comprises: distance information between the wave sensor systemand objects within the predetermined area.

According to the first aspect of the embodiments, the objects compriseone or more of a floor, table, walls, and other furniture.

According to the first aspect of the embodiments, the adaptivebeamforming circuit is further adapted to adapt one or more beams thattakes into account the distance information generated by the wave sensorsystem.

According to the first aspect of the embodiments, the adaptivebeamforming circuit is adapted to modify one or more of a beam width,beam reception angle, and range of the beam based on the receiveddistance information generated by the wave sensor system.

According to the first aspect of the embodiments, the adaptivebeamforming circuit is further adapted to receive the area image datasignal, the user location data signal, and the plurality of mic audiosignals, and perform adaptive beamforming on the plurality of mic audiosignals that takes into account the information in the area image datasignal and the user location data signal, such that the adaptivebeamforming circuit is further adapted to substantially ignore voicesignals that originate from outside the areas where the users arelocated.

According to the first aspect of the embodiments, the adaptivebeamforming circuit is further adapted to receive the area image datasignal, the user location data signal, and the plurality of mic audiosignals, and perform adaptive beamforming on the plurality of mic audiosignals that takes into account the information in the area image datasignal and the user location data signal, such that the adaptivebeamforming circuit is further adapted to substantially ignore audiosignals generated from one or more of a television and stereo.

According to the first aspect of the embodiments, the predetermined areais a conference room, there is at least one table located in theconference room, and further wherein the area image data signal includesinformation as to a location of the at least one table in the conferenceroom, and further wherein the adaptive beamforming circuit is adapted togenerate one or more fixed beam positions covering a perimeter of the atleast one table in the conference room.

According to the first aspect of the embodiments, the adaptivebeamforming circuit comprises: an acoustic audio direction of arrivalalgorithm adapted to determine direction of arrival of one or moremicrophone generated audio signals.

According to the first aspect of the embodiments, the direction ofarrival algorithm is adapted to determine a direction of arrival of theone or more microphone generated audio signals using information in thearea image data signal received from the wave sensor system.

According to the first aspect of the embodiments, the wave sensor systemcan determine motion of one or more objects located in the predeterminedarea.

According to the first aspect of the embodiments, the wave sensor systemcan include the object motion information about the predetermined areain the area image data signal, and wherein the adaptive beamformingcircuit can eliminate fixed objects and objects moving at asubstantially constant rate to determine a number of people located inthe predetermined area, and output the same as a room occupancy status.

According to the first aspect of the embodiments, the room occupancystatus can be used by other interconnected systems to control one ormore of lights, temperature, and audio-video equipment in the conferenceroom.

According to the first aspect of the embodiments, the room occupancystatus can be transmitted to a room monitoring system.

According to the first aspect of the embodiments, the predetermined areacomprises: a conference room.

According to the first aspect of the embodiments, the adaptivebeamforming circuit is further adapted to generate one or more beams toacquire sound from one or more specific locations in the predeterminedarea.

According to the first aspect of the embodiments, wherein thebeamforming microphone array further comprises: a plurality of acousticecho cancellation devices, one for each of the plurality of microphones,wherein each is adapted to receive the mic audio signal from arespective one of the plurality of microphones, perform acoustic echocancellation on the received mic audio signal, and output anecho-corrected mic audio signal.

According to the first aspect of the embodiments, the beamformingmicrophone array further comprises: a first communication device adaptedto receive a reference signal from a remote source, and forward the sameto each of the one or more acoustic echo cancellation devices, andwherein each of the one or more acoustic echo cancellation devices isadapted to delete the reference signal from a respective one of themicrophone audio signals received by the respective acoustic echocancellation devices.

According to the first aspect of the embodiments, the reference signalcomprises a far end audio signal.

According to the first aspect of the embodiments, the adaptivebeamforming circuit is adapted to adapt new beams no faster than a firstbeam formation rate, and wherein the acoustic echo cancellation deviceis adapted to perform echo cancellation no faster than a first echocancellation rate, and still further wherein the first echo cancellationrate and the first beam formation rate are substantially equivalent.

According to the first aspect of the embodiments, the wave sensor systemis adapted to resolve distances within the predetermined area withinabout 1 mm and within about 1 degree.

According to the first aspect of the embodiments, the predetermined areais a conference room, and the adaptive beamforming circuit is adapted toextract location information for each person in the conference room andis further adapted to adapt a respective fixed beam position for eachperson in the conference room.

According to the first aspect of the embodiments, the predetermined areais a conference room, and if the user location data signal indicatesthat there are more people than beams that can be formed, then theadaptive beamforming circuit is further adapted to modify one or more ofthe fixed beam positions to cover two or more people in the conferenceroom such that each person is covered by at least one fixed beam.

According to the first aspect of the embodiments, the adaptivebeamforming circuit is adapted to adjust a beam width and shape to covertwo or more people in the conference room.

According to the first aspect of the embodiments, the adaptivebeamforming circuit comprises: an automixer algorithm, and wherein theadaptive beamforming circuit is adapted to adapt multiple beams and thencombine the multiple beams to produce a single audio signal using theautomixer algorithm.

According to the first aspect of the embodiments, the beamformingmicrophone array further comprises: an active noise reduction circuitadapted to remove noise from an output of the adaptive beamformingcircuit, and output a noise reduced audio signal; an Ethernetcommunication device adapted to receive a far end audio signal from aremote location and output the same to one or more speakers and to eachof the acoustic echo cancellation devices, and wherein the Ethernetcommunication device is further adapted to receive as an input the noisereduced audio signal from the active noise reduction circuit, and outputthe same to the remote location; and a power-over-Ethernet deviceadapted to extract electrical power over Ethernet communications cablesand provide the electrical power to the circuits in the beamformingarray.

According to the first aspect of the embodiments, the beamformingmicrophone array further comprises: one or more of each of lightsensors, temperature sensors, and humidity sensors, and wherein thebeamforming microphone array is adapted to receive as inputs outputsfrom each of the sensors, and output the sensor outputs through theEthernet communication device.

According to the first aspect of the embodiments, the wave sensor systemis adapted to recognize gestures including one or more of hand motionand arm motion.

According to the first aspect of the embodiments, the recognizedgestures can control one or more functions in the conference room, andwherein the functions include one or more of lighting levels, audiolevels, temperature levels, humidity levels, and positions of shadesand/or curtains.

According to a second aspect of the embodiments, a beamformingmicrophone array is provided comprising: a plurality of microphones eachof which is adapted to receive an acoustic audio signal and convert thesame to a microphone (mic) audio signal; a wave sensor system adapted todetermine locations of one or more people within a predetermined areaabout the beamforming microphone array and output the same as userlocation data signal; and an adaptive beamforming circuit adapted toreceive the user location data signal and plurality of mic audio signalsand perform adaptive beamforming on the plurality of mic audio signalsthat takes into account the received user location data signal to adapta plurality of beam signals, one for each of the microphones, to acquiresound from one or more specific locations in the predetermined area; anda plurality of acoustic echo cancellation devices, one for each of thebeam signal outputs from the adaptive beamforming circuit, wherein eachof the plurality of acoustic echo cancellation devices is adapted toreceive a respective beam signal from the adaptive beamforming circuitand perform acoustic echo cancellation on the received respective beamsignal and output the echo-corrected beam signal.

According to the second aspect of the embodiments, the wave sensorsystem comprises: a millimeter (mm) wave transmitter; and a wavereceiver.

According to the second aspect of the embodiments, the wave sensorsystem comprises: an optical transmitter; and an optical receiver.

According to the second aspect of the embodiments, the wave sensorsystem is further adapted to generate a three dimensional image of thepredetermined area and output the same as an area image data signal.

According to the second aspect of the embodiments, the adaptivebeamforming circuit is further adapted to receive the area image datasignal and the plurality of mic audio signals and perform adaptivebeamforming on the plurality of mic audio signals that takes intoaccount the received area image data signal to adapt one or more beamsto acquire sound from one or more specific locations in thepredetermined area.

According to aspect of the embodiments, the adaptive beamforming circuitis adapted to modify the beam audio signals to reduce noise reflectedoff one or more objects within the predetermined area based on the areaimage data signal.

According to the second aspect of the embodiments, the area image datasignal comprises: information as to where motion is occurring within thepredetermined area.

According to the second aspect of the embodiments, the informationcontained within the area image data signal that motion is occurringwithin the predetermined area substantially eliminates objects that aresubstantially at rest.

According to the second aspect of the embodiments, the informationcontained within the area image data signal that motion is occurringwithin the predetermined area does not include objects that move with asubstantial constant velocity.

According to the second aspect of the embodiments the object that moveswith a substantially constant periodicity comprises a fan.

According to the second aspect of the embodiments, the area image datasignal comprises: distance information between the wave sensor systemand objects within the predetermined area.

According to the second aspect of the embodiments, the objects compriseone or more of a floor, table, walls, and other furniture.

According to the second aspect of the embodiments, the adaptivebeamforming circuit is further adapted to adapt one or more beams thattakes into account the distance information generated by the wave sensorsystem.

According to the second aspect of the embodiments, the adaptivebeamforming circuit is adapted to modify one or more of a beam width,beam reception angle, and range of the beam based on the receiveddistance information generated by the wave sensor system.

According to the second aspect of the embodiments, the adaptivebeamforming circuit is further adapted to receive the area image datasignal, the user location data signal, and the plurality of mic audiosignals, and perform adaptive beamforming on the plurality of mic audiosignals that takes into account the information in the area image datasignal and the user location data signal, such that the adaptivebeamforming circuit is further adapted to substantially ignore voicesignals that originate from outside the areas where the users arelocated.

According to the second aspect of the embodiments, the adaptivebeamforming circuit is further adapted to receive the area image datasignal, the user location data signal, and the plurality of mic audiosignals, and perform adaptive beamforming on the plurality of mic audiosignals that takes into account the information in the area image datasignal and the user location data signal, such that the adaptivebeamforming circuit is further adapted to substantially ignore audiosignals generated from one or more of a television and stereo.

According to the second aspect of the embodiments, the predeterminedarea is a conference room, there is at least one table located in theconference room, and further wherein the area image data signal includesinformation as to a location of the at least one table in the conferenceroom, and further wherein the adaptive beamforming circuit is adapted toadapt one or more fixed beam positions covering a perimeter of the atleast one table in the conference room.

According to the second aspect of the embodiments, the adaptivebeamforming circuit comprises: an acoustic audio direction of arrivalalgorithm adapted to determine direction of arrival of one or moremicrophone generated audio signals.

According to the second aspect of the embodiments, the direction ofarrival algorithm is adapted to determine a direction of arrival of theone or more microphone generated audio signals using information in thearea image data signal received from the wave sensor system.

According to the second aspect of the embodiments, the wave sensorsystem can determine motion of one or more objects located in thepredetermined area.

According to the second aspect of the embodiments, the wave sensorsystem can include the object motion information about the predeterminedarea in the area image data signal, and wherein the adaptive beamformingcircuit can eliminate fixed objects and objects moving at asubstantially constant rate to determine a number of people located inthe predetermined area, and output the same as a room occupancy status.

According to the second aspect of the embodiments, the room occupancystatus can be used by other interconnected systems to control one ormore of lights, temperature, and audio-video equipment in the conferenceroom.

According to the second aspect of the embodiments, the room occupancystatus can be transmitted to a room monitoring system.

According to the second aspect of the embodiments, the predeterminedarea comprises: a conference room.

According to the second aspect of the embodiments, the adaptivebeamforming circuit is further adapted to generate one or more beams toacquire sound from one or more specific locations in the predeterminedarea.

According to the second aspect of the embodiments, the adaptivebeamforming circuit is adapted to generate new beams no faster than afirst beam formation rate, and wherein the acoustic echo cancellationdevice is adapted to perform echo cancellation no faster than a firstecho cancellation rate, and still further wherein the first echocancellation rate and the first beam formation rate are substantiallyequivalent.

According to the second aspect of the embodiments, the wave sensorsystem is adapted to resolve distances within the predetermined areawithin about 1 mm and within about 1 degree.

According to the second aspect of the embodiments, the predeterminedarea is a conference room, and the adaptive beamforming circuit isadapted to extract location information for each person in theconference room and generate a respective fixed beam position for eachperson in the conference room.

According to the second aspect of the embodiments, the predeterminedarea is a conference room, and if the user location data signalindicates that there are more people than beams that can be formed, thenthe adaptive beamforming circuit is further adapted to modify one ormore of the fixed beam positions to cover two or more people in theconference room such that each person is covered by at least one fixedbeam.

According to the second aspect of the embodiments, the adaptivebeamforming circuit is adapted to adjust a beam width and shape to covertwo or more people in the conference room.

According to the second aspect of the embodiments, the adaptivebeamforming circuit further comprises: a plurality of active noisereduction circuits, one for each acoustic echo cancellation device, andwherein each of the active noise reduction circuits is adapted to removenoise from an output of its respective acoustic echo cancellation deviceand output a noise reduced audio signal; an N-1 auto-mixer deviceadapted to receive the plurality of noise reduced audio signals from theplurality of active noise reductions circuits and combine the pluralityof noise reduced audio signals to output a single near end audio signal;and

an Ethernet communication device adapted to receive a reference signalfrom a remote source, output the same to one or more speakers in thepredetermined area, and forward the same to each of the one or moreacoustic echo cancellation devices, and wherein each of the one or moreacoustic echo cancellation devices is adapted to delete the referencesignal from a respective one of the microphone audio signals received bythe respective acoustic echo cancellation devices; and apower-over-Ethernet device adapted to extract electrical power overEthernet communications cables and provide the electrical power to thecircuits in the beamforming array.

According to the second aspect of the embodiments, the reference signalcomprises a far end audio signal.

According to the second aspect of the embodiments, the beamformingmicrophone array further comprises: one or more of each of lightsensors, temperature sensors, and humidity sensors, and wherein thebeamforming microphone array is adapted to receive as inputs outputsfrom each of the sensors, and output the sensor outputs through theEthernet communication device.

According to the second aspect of the embodiments, the wave sensorsystem is adapted to recognize gestures including one or more of handmotion and arm motion.

According to the second aspect of the embodiments, the recognizedgestures can control one or more functions in the conference room, andwherein the functions include one or more of lighting levels, audiolevels, temperature levels, humidity levels, and positions of shadesand/or curtains.

According to a third aspect of the embodiments, a method for operating abeamforming microphone array for use in a predetermined area isprovided, the method comprising: receiving acoustic audio signals ateach of a plurality of microphones, converting the same to an electricalmic audio signal, and outputting each of the plurality of electrical micaudio signals; generating a user location data signal by a wave sensorsystem, and outputting the user location data signal, wherein the userlocation data signal includes location information of one or more peoplewithin the predetermined area; receiving both the user location datasignal and plurality of echo-corrected mic audio signals at an adaptivebeamforming device; and adapting one or more beams by the adaptivebeamforming device based on the user location data signal and pluralityof mic audio signals wherein each of the one or more beams acquiressound from one or more specific locations in the predetermined area.

According to the third aspect of the embodiments, the wave sensor systemcomprises: a millimeter (mm) wave transmitter; and a wave receiver.

According to the third aspect of the embodiments, the wave sensor systemcomprises: an optical transmitter; and an optical receiver.

According to the third aspect of the embodiments, the method furthercomprises: generating a three dimensional image of the predeterminedarea by the wave sensor system; and outputting the same as an area imagedata signal.

According to the third aspect of the embodiments, the method furthercomprises: receiving the area image data signal and the plurality of micaudio signals by the adaptive beamforming circuit; performing adaptivebeamforming on the plurality of mic audio signals that takes intoaccount the received area image data signal and the plurality of micaudio signals; and adapting one or more beams to acquire sound from oneor more specific locations in the predetermined area.

According to the third aspect of the embodiments, the method furthercomprises: modifying the beam audio signals by the adaptive beamformingcircuit to reduce noise reflected off one or more objects within thepredetermined area based on the area image data signal.

According to the third aspect of the embodiments, the area image datasignal comprises: information as to where motion is occurring within thepredetermined area. According to the third aspect of the embodiments,the information contained within the area image data signal that motionis occurring within the predetermined area substantially eliminatesobjects that are substantially at rest.

According to the third aspect of the embodiments, the informationcontained within the area image data signal that motion is occurringwithin the predetermined area does not include objects that move with asubstantial constant velocity.

According to the third aspect of the embodiments, the object that moveswith a substantially constant periodicity comprises a fan.

According to the third aspect of the embodiments, the area image datasignal comprises: distance information between the wave sensor systemand objects within the predetermined area.

According to the third aspect of the embodiments, the objects compriseone or more of a floor, table, walls, and other furniture.

According to the third aspect of the embodiments, the method furthercomprises: adapting one or more beams by the adaptive beamformingcircuit that takes into account the distance information generated bythe wave sensor system.

According to the third aspect of the embodiments, the method furthercomprises: modifying, by the adaptive beamforming circuit, one or moreof a beam width, beam reception angle, and range of the beam based onthe received distance information generated by the wave sensor system.

According to the third aspect of the embodiments, the method furthercomprises: receiving, by the adaptive beamforming circuit, the areaimage data signal, the user location data signal, and the plurality ofmic audio signals; and performing adaptive beamforming on the pluralityof mic audio signals that takes into account the information in the areaimage data signal and the user location data signal, such that theadaptive beamforming circuit substantially ignores voice signals thatoriginate from outside the areas where the users are located.

According to the third aspect of the embodiments, the method furthercomprises: receiving, by the adaptive beamforming circuit, the areaimage data signal, the user location data signal, and the plurality ofmic audio signals; and performing adaptive beamforming on the pluralityof mic audio signals that takes into account the information in the areaimage data signal and the user location data signal, such that theadaptive beamforming circuit substantially ignore audio signalsgenerated from one or more of a television and stereo.

According to the third aspect of the embodiments, the predetermined areais a conference room, there is at least one table located in theconference room, and further wherein the area image data signal includesinformation as to a location of the at least one table in the conferenceroom, and the method further comprises: generating, by the adaptivebeamforming circuit, one or more fixed beam positions covering aperimeter of the at least one table in the conference room.

According to the third aspect of the embodiments, the method furthercomprises: determining, by an acoustic audio direction of arrivalalgorithm operating within the adaptive beamforming circuit, a directionof arrival of one or more microphone generated audio signals.

According to the third aspect of the embodiments, the method furthercomprises: determining, by the direction of arrival algorithm, adirection of arrival of the one or more microphone generated audiosignals using information in the area image data signal received fromthe wave sensor system.

According to the third aspect of the embodiments, the method furthercomprises: determining, by the wave sensor system, motion of one or moreobjects located in the predetermined area.

According to the third aspect of the embodiments, the wave sensor systemcan include the object motion information about the predetermined areain the area image data signal, and wherein the adaptive beamformingcircuit can eliminate fixed objects and objects moving at asubstantially constant rate to determine a number of people located inthe predetermined area, and output the same as a room occupancy status.

According to the third aspect of the embodiments, the room occupancystatus can be used by other interconnected systems to control one ormore of lights, temperature, and audio-video equipment in the conferenceroom.

According to the third aspect of the embodiments, the room occupancystatus can be transmitted to a room monitoring system.

According to the third aspect of the embodiments, the predetermined areacomprises: a conference room.

According to the third aspect of the embodiments, the method furthercomprises: generating, by the adaptive beamforming circuit, one or morebeams to acquire sound from one or more specific locations in thepredetermined area.

According to the third aspect of the embodiments, the method furthercomprises: receiving, by a plurality of acoustic echo cancellationdevices, one for each of the plurality of microphones, the mic audiosignal from a respective one of the plurality of microphones; performingacoustic echo cancellation on the received mic audio signal; andoutputting an echo-corrected mic audio signal.

According to the third aspect of the embodiments, the method furthercomprises: receiving, by a first communication device adapted to receivea reference signal from a remote source, and forward the same to each ofthe one or more acoustic echo cancellation devices, and wherein each ofthe one or more acoustic echo cancellation devices is adapted to deletethe reference signal from a respective one of the microphone audiosignals received by the respective acoustic echo cancellation devices.

According to the third aspect of the embodiments, the reference signalcomprises a far end audio signal.

According to the third aspect of the embodiments, the method furthercomprises: the adaptive beamforming circuit adapting to new beams nofaster than a first beam formation rate; and the acoustic echocancellation device performing echo cancellation no faster than a firstecho cancellation rate, and wherein the first echo cancellation rate andthe first beam formation rate are substantially equivalent.

According to the third aspect of the embodiments, the wave sensor systemis adapted to resolve distances within the predetermined area withinabout 1 mm and within about 1 degree.

According to the third aspect of the embodiments, the predetermined areais a conference room, and the method further comprises: extracting, bythe adaptive beamforming circuit, location information for each personin the conference room, and adapting a respective fixed beam positionfor each person in the conference room.

According to the third aspect of the embodiments, the predetermined areais a conference room, and if the user location data signal indicatesthat there are more people than beams that can be formed, thenmodifying, by the adaptive beamforming circuit, one or more of the fixedbeam positions to cover two or more people in the conference room suchthat each person is covered by at least one fixed beam.

According to the third aspect of the embodiments, the method furthercomprises: adjusting, by the adaptive beamforming circuit, a beam widthand shape to cover two or more people in the conference room.

According to the third aspect of the embodiments, the adaptivebeamforming circuit comprises: an automixer algorithm, and wherein themethod further comprises adapting, by the adaptive beamforming circuit,multiple beams and combing the multiple beams to produce a single audiosignal using the automixer algorithm.

According to the third aspect of the embodiments, the method furthercomprises: removing, by an active noise reduction circuit, noise from anoutput of the adaptive beamforming circuit, and outputting a noisereduced audio signal; receiving, by an Ethernet communication device, afar end audio signal from a remote location and outputting the same toone or more speakers and to each of the acoustic echo cancellationdevices; receiving, by the Ethernet communication device, as an inputthe noise reduced audio signal from the active noise reduction circuit,and outputting the same to the remote location; and extracting, by apower-over-Ethernet device, electrical power over Ethernetcommunications cables and providing the electrical power to the circuitsin the beamforming array.

According to the third aspect of the embodiments, the predetermined areafurther comprises: one or more of each of light sensors, temperaturesensors, and humidity sensors, and wherein the method further comprisesreceiving, by the beamforming microphone array, as inputs the outputsfrom each of the sensors, and outputting the sensor outputs through theEthernet communication device.

According to the third aspect of the embodiments, the method furthercomprises: recognizing, by the wave sensor system, gestures includingone or more of hand motion and arm motion.

According to the third aspect of the embodiments, the recognizedgestures can control one or more functions in the conference room, andwherein the functions include one or more of lighting levels, audiolevels, temperature levels, humidity levels, and positions of shadesand/or curtains.

According to the fourth aspect of the embodiments,

According to a fourth aspect of the embodiments, a method for operatinga beamforming microphone array for use in a predetermined area isprovided, comprising: receiving acoustic audio signals at each of aplurality of microphones, converting the same to an electrical mic audiosignal, and outputting each of the plurality of electrical mic audiosignals; generating a user location data signal by a wave sensor system,and outputting the user location data signal, wherein the user locationdata signal includes location information of one or more people withinthe predetermined area; receiving both the user location data signal andplurality of mic audio signals at an adaptive beamforming device;adapting one or more beams by the adaptive beamforming device based onthe user location data signal and plurality of output electrical micaudio signals wherein each of the one or more beams acquires sound fromone or more specific locations in the predetermined area; and performingacoustic echo cancellation on each of the one or more beams output fromthe adaptive beamforming device.

According to the fourth aspect of the embodiments, the wave sensorsystem comprises: a millimeter (mm) wave transmitter; and a wavereceiver.

According to the fourth aspect of the embodiments, the wave sensorsystem comprises: an optical transmitter; and an optical receiver.

According to the fourth aspect of the embodiments, the method furthercomprises: generating, by the wave sensor system, a three dimensionalimage of the predetermined area and output the same as an area imagedata signal.

According to the fourth aspect of the embodiments, the method furthercomprises: receiving, by the adaptive beamforming circuit, the areaimage data signal and the plurality of mic audio signals; and performingadaptive beamforming on the plurality of mic audio signals that takesinto account the received area image data signal to adapt one or morebeams to acquire sound from one or more specific locations in thepredetermined area.

According to the fourth aspect of the embodiments, the method furthercomprises: modifying, by the adaptive beamforming circuit, the beamaudio signals to reduce noise reflected off one or more objects withinthe predetermined area based on the area image data signal.

According to the fourth aspect of the embodiments, the area image datasignal comprises: information as to where motion is occurring within thepredetermined area.

According to the fourth aspect of the embodiments, the informationcontained within the area image data signal that motion is occurringwithin the predetermined area substantially eliminates objects that aresubstantially at rest.

According to the fourth aspect of the embodiments, the informationcontained within the area image data signal that motion is occurringwithin the predetermined area does not include objects that move with asubstantial constant velocity.

According to the fourth aspect of the embodiments, the object that moveswith a substantially constant periodicity comprises a fan.

According to the fourth aspect of the embodiments, the area image datasignal comprises: distance information between the wave sensor systemand objects within the predetermined area.

According to the fourth aspect of the embodiments, the objects compriseone or more of a floor, table, walls, and other furniture.

According to the fourth aspect of the embodiments, the method furthercomprises: adapting, by the adaptive beamforming circuit, one or morebeams that takes into account the distance information generated by thewave sensor system.

According to the fourth aspect of the embodiments, the method furthercomprises: modifying, by the adaptive beamforming circuit, one or moreof a beam width, beam reception angle, and range of the beam based onthe received distance information generated by the wave sensor system.

According to the fourth aspect of the embodiments, the method furthercomprises; receiving, by the adaptive beamforming circuit, the areaimage data signal, the user location data signal, and the plurality ofmic audio signals; and performing adaptive beamforming on the pluralityof mic audio signals that takes into account the information in the areaimage data signal and the user location data signal, such that theadaptive beamforming circuit is further adapted to substantially ignorevoice signals that originate from outside the areas where the users arelocated.

According to the fourth aspect of the embodiments, the method furthercomprises: receiving, by the adaptive beamforming circuit, the areaimage data signal, the user location data signal, and the plurality ofmic audio signals; and performing adaptive beamforming on the pluralityof mic audio signals that takes into account the information in the areaimage data signal and the user location data signal, such that theadaptive beamforming circuit is further adapted to substantially ignoreaudio signals generated from one or more of a television and stereo.

According to the fourth aspect of the embodiments, the predeterminedarea is a conference room, there is at least one table located in theconference room, and further wherein the area image data signal includesinformation as to a location of the at least one table in the conferenceroom, and wherein the method further comprises: adapting, by theadaptive beamforming circuit, one or more fixed beam positions to covera perimeter of the at least one table in the conference room.

According to the fourth aspect of the embodiments, the method furthercomprises: determining a direction of arrival of one or more microphonegenerated audio signals by an acoustic audio direction of arrivalalgorithm stored with the adaptive beamforming circuit.

According to the fourth aspect of the embodiments, the method furthercomprises: determining the direction of arrival of the one or moremicrophone generated audio signals, in the adaptive beamforming circuit,using information in the area image data signal received from the wavesensor system.

According to the fourth aspect of the embodiments, the method furthercomprises: determining, by the wave sensor system, motion of one or moreobjects located in the predetermined area.

According to the fourth aspect of the embodiments, the wave sensorsystem can include the object motion information about the predeterminedarea in the area image data signal, and wherein the method furthercomprises: substantially eliminating, by the adaptive beamformingcircuit, fixed objects and objects moving at a substantially constantrate to determine a number of people located in the predetermined area,and output the same as a room occupancy status.

According to the fourth aspect of the embodiments, the method furthercomprises: using the room occupancy status by other interconnectedsystems to control one or more of lights, temperature, and audio-videoequipment in the conference room.

According to the fourth aspect of the embodiments, the method furthercomprises: transmitting the room occupancy status to a room monitoringsystem.

According to the fourth aspect of the embodiments, the predeterminedarea comprises: a conference room.

According to the fourth aspect of the embodiments, the method furthercomprises: generating, by the adaptive beamforming circuit, one or morebeams to acquire sound from one or more specific locations in thepredetermined area.

According to the fourth aspect of the embodiments, the method furthercomprises: receiving, by a first communication device, a referencesignal from a remote source; forwarding the reference signal to each ofthe one or more of the acoustic echo cancellation devices; and deleting,by each of the one or more acoustic echo cancellation devices, thereference signal from a respective one of the microphone audio signalsreceived by the respective acoustic echo cancellation devices.

According to the fourth aspect of the embodiments, wherein the referencesignal comprises a far end audio signal.

According to the fourth aspect of the embodiments, the method furthercomprises: generating, by the adaptive beamforming circuit, new beams nofaster than a first beam formation rate; and performing, by the acousticecho cancellation device, echo cancellation no faster than a first echocancellation rate, and still further wherein the first echo cancellationrate and the first beam formation rate are substantially equivalent.

According to the fourth aspect of the embodiments, wherein the wavesensor system is adapted to resolve distances within the predeterminedarea within about 1 mm and within about 1 degree.

According to the fourth aspect of the embodiments, wherein thepredetermined area is a conference room, and wherein the method furthercomprises: extracting, by the adaptive beamforming circuit, locationinformation for each person in the conference room; and generating arespective fixed beam position for each person in the conference room.

According to the fourth aspect of the embodiments, wherein thepredetermined area is a conference room, and wherein the method furthercomprises: modifying, by the adaptive beamforming circuit, if the userlocation data signal indicates that there are more people than beamsthat can be formed, one or more of the fixed beam positions to cover twoor more people in the conference room such that each person is coveredby at least one fixed beam.

According to the fourth aspect of the embodiments, the method furthercomprises: adjusting, by the adaptive beamforming circuit, a beam widthand shape to cover two or more people in the conference room.

According to the fourth aspect of the embodiments, the method furthercomprises: removing noise, by a plurality of active noise reductioncircuits, one for each acoustic echo cancellation device, from an outputof its respective acoustic echo cancellation device; outputting a noisereduced audio signal; receiving, by an N-1 auto-mixer device, theplurality of noise reduced audio signals from the plurality of activenoise reductions circuits; combining the plurality of noise reducedaudio signals to output a single near end audio signal; and receiving,by an Ethernet communication device, a reference signal from a remotesource; outputting the reference signal to one or more speakers in thepredetermined area; forwarding the reference signal to each of the oneor more acoustic echo cancellation devices; deleting, by the acousticecho cancellation device, the reference signal from a respective one ofthe microphone audio signals received by the respective acoustic echocancellation devices; and extracting, by a power-over-Ethernet device,electrical power over one or more Ethernet communications cables andproviding the electrical power to the circuits in the beamformingmicrophone array.

According to the fourth aspect of the embodiments, the reference signalcomprises a far end audio signal.

According to the fourth aspect of the embodiments, the method furthercomprises: outputting, by one or more of each of light sensors,temperature sensors, and humidity sensors, status outputs from each ofthe sensors; receiving the sensors outputs by the Ethernetcommunications device; and transmitting, by the Ethernet communicationsdevice, the sensor outputs.

According to the fourth aspect of the embodiments, the method furthercomprises: recognizing, by the wave sensor system, gestures includingone or more of hand motion and arm motion.

According to the fourth aspect of the embodiments, the recognizedgestures can control one or more functions in the conference room, andwherein the functions include one or more of lighting levels, audiolevels, temperature levels, humidity levels, and positions of shadesand/or curtains.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and features of the embodiments will becomeapparent and more readily appreciated from the following description ofthe embodiments with reference to the following figures. Differentaspects of the embodiments are illustrated in reference figures of thedrawings. It is intended that the embodiments and figures disclosedherein are to be considered to be illustrative rather than limiting. Thecomponents in the drawings are not necessarily drawn to scale, emphasisinstead being placed upon clearly illustrating the principles of theaspects of the embodiments. In the drawings, like reference numeralsdesignate corresponding parts throughout the several views.

FIG. 1 illustrates a top view of a room that uses a knownaudio-conferencing system.

FIG. 2 illustrates a block diagram of an audio processing systemcharacterized as using an adaptive beamformer with acoustic echocancellation before beamforming for use with the room and conferencingsystem shown in FIGS. 1 and 4 that comprises an acoustic echocancellation device (AECD), an adaptive beamforming circuit, and amillimeter wave sensor, among other devices, according to aspects of theembodiments.

FIG. 3 illustrates a block diagram of an audio processing systemcharacterized as using an adaptive beamformer with acoustic echocancellation following beamforming for use with the room andconferencing system shown in FIGS. 1 and 4 that comprises an acousticecho cancellation device (AECD), an adaptive beamforming circuit, activenoise reduction circuitry, and a millimeter wave sensor, among otherdevices, according to aspects of the embodiments.

FIG. 4 illustrates a top view of a room substantially similar that ofFIG. 1, but which can include the audio processing systems of FIGS. 2and 3 according to aspects of the embodiments.

FIG. 5 illustrates a flow chart of a method for operating a beamformingmicrophone array in a conference room in which a plurality ofmicrophones output audio signals to an equal plurality of acoustic echocancellation devices that provide a plurality of echo cancelled audiosignals to an adaptive beamforming device that receives as an input aroom image signal to facilitate generation of an audio beam signal thesubject to active noise reduction prior to be sent to a far endconference room according to aspects of the embodiments.

FIG. 6 illustrates a flow chart of a method for operating a beamformingmicrophone array in a conference room in which a plurality ofmicrophones output audio signals to an adaptive beamforming device thatreceives as an input a room image signal to facilitate generation of aplurality of audio beam signals that are then processed individually byacoustic echo cancellation devices and active noise reductions circuityprior to being auto-mixed and sent to a far end conference roomaccording to aspects of the embodiments.

FIG. 7 illustrates a personal computer/processor/laptop suitable for useto implement the methods shown in FIGS. 5 and 6, among other methods,for optimizing adaptive beamforming according to aspects of theembodiments.

FIG. 8 illustrates a network system within which the systems and methodsshown in FIGS. 1-6 can operate for optimizing adaptive beamformingaccording to aspects of the embodiments.

DETAILED DESCRIPTION

The embodiments are described more fully hereinafter with reference tothe accompanying drawings, in which embodiments of the inventive conceptare shown. Like numbers refer to like elements throughout. Theembodiments may, however, be embodied in many different forms and shouldnot be construed as limited to the embodiments set forth herein. Rather,these embodiments are provided so that this disclosure will be thoroughand complete, and will fully convey the scope of the inventive conceptto those skilled in the art. The scope of the embodiments is thereforedefined by the appended claims. The detailed description that follows iswritten from the point of view of a control systems company, so it is tobe understood that generally the concepts discussed herein areapplicable to various subsystems and not limited to only a particularcontrolled device or class of devices, such as audio systems and relateddevices, audio-networking devices, and mechanical systems related toaudio systems and devices.

Reference throughout the specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with an embodiment is included inat least one embodiment of the embodiments. Thus, the appearance of thephrases “in one embodiment” or “in an embodiment” in various placesthroughout the specification is not necessarily referring to the sameembodiment. Further, the particular feature, structures, orcharacteristics may be combined in any suitable manner in one or moreembodiments.

List of Reference Numbers for the Elements in the Drawings in NumericalOrder

The following table is a list of the major elements in the drawings innumerical order.

-   100 Room-   104 People-   106 Table-   108 Speaker/Microphone Assembly-   110 Conference System Processor (Processor)-   112 Chair-   114 Audio/Video (NV) Display-   116 Door-   118 Wall-   150 Audio Conferencing System (ACS)-   200 Audio Processing System-   202 Microphone (Mic)-   204 Acoustic Echo Cancellation Device (AECD)-   206 Adaptive Beamforming Circuitry-   208 Active Noise Reduction (ANR) Circuit-   210 Network (NW) Interface (I/F) (Audio Conference Computer)-   212 Millimeter (mm) Wave Antenna (MWA)-   214 Millimeter Wave Transceiver (MWT)-   216 Power-over-Ethernet (PoE) Converter-   218 Reference Line-   220 Processor-   222 Memory-   224 Acoustic Echo Cancellation Software/Application (AEC App)-   226 Adaptive Beamforming Software/Application (ABF App)-   228 Active Noise Reduction Software/Application (ANR App)-   230 Conference System Software/Application (CS App)-   232 Network (Internet)-   250 Wave Sensor System (WSS)-   300 Audio Processing System-   302 Auto Mixer-   500 Method for Operating Audio Processor System 200-   502-516 Method Steps in Method 500-   600 Method for Operating Audio Processor System 300-   602-618 Method Steps in Method 600-   701 Shell/Box-   702 Integrated Display/Touch-Screen (Display)-   704 Internal Data/Command Bus (Bus)-   706 Processor Internal Memory-   710 Universal Serial Bus (USB) Port-   711 Ethernet Port-   712 Compact Disk (CD)/Digital Video Disk (DVD) Read/Write (RW)    (CD/DVD/RW) Drive-   714 Floppy Diskette Drive-   716 Hard Disk Drive (HDD)-   718 Read-Only Memory (ROM)-   720 Random Access Memory (RAM)-   722 Video Graphics Array (VGA) Port or High Definition Multimedia    Interface (HDMI)-   723 HDMI Cable-   724 External Memory Storage Device-   726 External Display/Touch-Screen (External Display)-   728 Keyboard-   730 Mouse-   732 Processor Board/PC Internal Memory (Internal Memory)-   734 Flash Drive Memory-   736 CD/DVD Diskettes-   738 Floppy Diskettes-   742 Wi-Fi Transceiver-   744 BlueTooth (BT) Transceiver-   746 Near Field Communications (NFC) Transceiver-   748 Third Generation (3G), Fourth Generation (4G), Long Term    Evolution (LTE) (3G/4G/LTE) Transceiver-   750 Communications Satellite/Global Positioning System (Satellite)    Transceiver Device-   752 Antenna-   756 Universal Serial Bus (USB) Cable-   758 Ethernet Cable (CATS)-   760 Scanner/Printer/Fax Machine-   800 Network System-   802 Mobile Device-   804 Personal Computer (PC)-   806 Internet Service Provider (ISP)-   808 Modulator/Demodulator (modem)-   810 Wireless Router (WiFi)-   812 Plain Old Telephone Service (POTS) Provider-   814 Cellular Service Provider-   818 Communications Satellite-   820 Cellular Tower-   824 GPS Station-   826 Satellite Communication Systems Control Stations-   828 Global Positioning System (GPS) Satellite

List of Acronyms Used in the Specification in Alphabetical Order

The following is a list of the acronyms used in the specification inalphabetical order.

-   3G Third Generation-   4G Fourth Generation-   ACS Audio Conferencing System-   AEC Acoustic Echo Cancellation-   AECD Acoustic Echo Cancellation Device-   ANR Active Noise Reduction-   App Application-   ARM Advanced Reduced Instruction Set Computer Machines-   ASIC Application Specific Integrated Circuitry-   A/V Audio/Video-   BIOS Basic Input/Output System-   BT BlueTooth-   CD Compact Disk-   CRC Cyclic Redundancy Check-   CRT Cathode Ray Tubes-   DSP Digital Signal Processor-   DVD Digital Video/Versatile Disk-   EEPROM Electrically Erasable Programmable Read Only Memory-   FE Far End-   FEC Forward Error Correction-   FPGA Field Programmable Gate Array Structures-   GAN Global Area Network-   GPS Global Positioning System-   HDD Hard Disk Drive-   HDMI High Definition Multimedia Interface-   HVAC Heating Ventilation and Air Conditioning-   Hz Hertz-   I2S Inter-Integrated Circuit Sound-   I/F Interface-   IP Internet Protocol-   ISP Internet Service Provider-   KHz Kilo-Hertz-   LCD Liquid Crystal Display-   LED Light Emitting Diode Display-   LSB Least Significant Bit-   LTE Long Term Evolution-   Mic Microphone-   MIPS Mega Instructions-Per-Second-   MODEM Modulator-Demodulator-   MSB Most Significant Bit-   Msec Millisecond-   MWA Millimeter Wave Antenna-   MWT Millimeter Wave Transceiver-   NFC Near Field Communication-   NLP Non-linear Processing-   N/W Network-   PC Personal Computer-   POTS Plain Old Telephone Service-   PTP Precision Time Protocol-   RAM Random Access Memory-   RISC Reduced Instruction Set Computer-   ROM Read Only Memory-   RW Read/Write-   SIMD Single Instructor Multiple Data-   SNR Signal-to-Noise Ratio-   TDM Time Division Multiplexing-   USB Universal Serial Bus-   UVPROM Ultra-violet Erasable Programmable Read Only Memory-   VGA Video Graphics Array-   WSS Wave Sensor System

The different aspects of the embodiments described herein pertain to thecontext of systems, methods, and modes for implementing a millimeterwave sensor to optimize operation of a beamforming microphone array, aswell as for use in other home or enterprise systems, but is not limitedthereto, except as may be set forth expressly in the appended claims.

For 40 years Creston Electronics Inc., has been the world's leadingmanufacturer of advanced control and automation systems, innovatingtechnology to simplify and enhance modern lifestyles and businesses.Crestron designs, manufactures, and offers for sale integrated solutionsto control audio, video, computer, and environmental systems. Inaddition, the devices and systems offered by Crestron streamlinestechnology, improving the quality of life in commercial buildings,universities, hotels, hospitals, and homes, among other locations.Accordingly, the systems, methods, and modes of the aspects of theembodiments described herein, as further embodied in the attacheddrawings, can be manufactured by Crestron Electronics Inc., located inRockleigh, N.J., and will be marketed and sold.

Aspects of the embodiments are directed towards systems, methods, andmodes for implementing a millimeter wave sensor to optimize operation ofa beamforming microphone array. Aspects of the embodiments can reducethe setup time for a mic array and improve mic array performance.Aspects of the embodiments can improve beam position and area coveragefor better voice pickup, which results in increased voiceintelligibility.

According to aspects of the embodiments, use of a millimeter wave sensor(or radar) allows for substantially automatic and periodic adjustment ofan adapting beamforming microphone that then also improves theperformance of beamforming while reducing the setup time and skillrequired.

As those of skill in the art can appreciate, millimeter waves are in the30 to 300 GHz spectrum. Sensors exist that work in a subset of this band(60 to 81 GHz) and implement a radar functionality. According to aspectsof the embodiments, transceivers can be utilized that incorporatesubstantially all of most of the functional requirements for such radarfunctions. As those of skill in the art can appreciate, such radarfunctionality can include a transmitter capable of transmitting a chirpmodulated waveform. A chirp modulated waveform is one in which thefrequency changes, either increasing or decreasing, typically in eithera linear or exponential manner (or some other manner), between a firstvalue to a second value.

As those of skill in the art can appreciate, a radar system uses a shortburst of radio frequency energy that is emitted from a directionalantenna. Objects reflect some of this energy back to a radio receiverlocated next to the transmitter. Since radio waves travel at a constantrate, the elapsed time between the transmitted and received signalsprovides the distance to the target. This brings up the firstrequirement for the pulse: it needs to be as short as possible. Forexample, a 1-microsecond pulse provides a radio burst about 300 meterslong. This means that the distance information we obtain with the systemwill have a resolution of about this same length. If we want betterdistance resolution, we need a shorter pulse.

The second requirement is that in order to detect objects farther away,more energy is needed in the transmitted pulse. Unfortunately, moreenergy and shorter pulse are conflicting requirements. The electricalpower needed to provide a pulse is equal to the energy of the pulsedivided by the pulse length. Requiring both more energy and a shorterpulse makes electrical power handling a limiting factor in the system.The output stage of a radio transmitter can only handle so much powerwithout destroying itself.

Chirp signals provide a way of breaking this limitation. Before theimpulse reaches the final stage of the radio transmitter, it is passedthrough a chirp system. Instead of bouncing an impulse off the targetaircraft, a chirp signal is used. After the chirp echo is received, thesignal is passed through an anti-chirp system, restoring the signal toan impulse. This allows the portions of the system that measure distanceto see short pulses, while the power handling circuits see long durationsignals. This type of wave-shaping is a fundamental part of modern radarsystems. As those of skill in the art can further appreciate, decreasingthe amount of power transmitted in small room that will have people init is generally preferable. Thus, even when used in rooms, transmittingchirp signals are a preferred means of object detection and resolution.According to further aspects of the embodiments, the transceiver furtherincludes a receiver that receives and demodulates the received signal.Subsequently, a three-dimensional image of the room can be determinedfrom the received returns.

FIG. 1 illustrates a top view of room 100 that uses a knownaudio-conferencing system (ACS) 150. ACS 150 comprises combined speakerand microphone (mic) 108, and conference system processor 110. In room100 there is located a plurality of people 104 a-c, with two such people104 d,e standing just outside room 100 near doorway 116. Also, in room100 there are numerous chairs 112, display 114 (which may or not be tiedinto ACS 150), table 106, and walls and ceiling 118. Depending on thedirectionality of combined mics/speakers 108 a,b there can one or moreaudio dropout zones, wherein it is more difficult to hear the audiooutput from the speaker, or to be heard by the mic. Such ACS 150 canalso pick up conversations outside of room 100 at doorway 116, even withthe use of beamforming.

FIG. 2 illustrates a block diagram of audio processing system (APS) 200characterized as using an adaptive beamformer with acoustic echocancellation before beamforming for use in a room and anaudio-conferencing system according to aspects of the embodiments. APS200 and APS 300, both of which are described below, can be used in theroom as embodied in FIG. 4: FIG. 4 illustrates a top view of a roomsubstantially similar that of FIG. 1, but which can include APSs 200,300 of FIGS. 2 and 3, respectively, according to aspects of theembodiments. APS 200 comprises mic array 203 that comprises mics 202a-m, acoustic echo cancellation devices (AECD) 204 a-m, adaptivebeamforming circuit 206, active noise reduction (ANR) circuit 208,Ethernet network interface (NW IF) 210 (which can also be referred to asa “audio conference computer” and is discussed in greater detail inregard to FIGS. 7 and 8, and which is connected to network 232 (which,in a non-limiting example, is the Internet, but which can also be manyother different types of networks), millimeter (mm) wave antenna (MWA)212, and millimeter wave transceiver (MWT) 214, among other devices,according to aspects of the embodiments. MWA 212 and MWT 214 comprisewave sensor system (WSS) 250. Also included is power-over-Ethernet (PoE)converter 216. The far end audio signal is introduced to AECDs 204 a-mvia reference line 218. According to further aspects of the embodiments,WSS 250 can be an optical-based sensor system, wherein transceiver 214would include an optical transmitter and receiver, and according tofurther aspects transceiver 214 can include a laser or infraredtransmitter and receiver.

APS 200, as discussed above, is an audio processing system that can becharacterized as an adaptive beamformer with AEC before beamforming.According to aspects of the embodiments, beamforming does not degradethe AEC in this case.

As those of skill in the art can further appreciate, the shortwavelength of the transmitted signals of WSS 250 allows for smallantennas (antennas are typically sized in terms of the wavelengths ofthe main frequency they are intended to transmit; in this case, MWA212). Using WSS 250 also provides high precision to resolve distances inthe mm range and angles down to about 1 degree (note that the resolutionis also dictated by the length of the transmitted pulse). In addition torange and angle, object velocity can be calculated by WSS 250. Anexample of this technology is the Texas Instruments IWR1642 single chipmillimeter (mm) wave sensor with integrating digital signal processor(DSP) and micro-controller unit (MCU).

Information from WSS 250 can be used to locate walls, floor, furnitureand people, among other objects, within a space. People can be detectedby detecting motion with certain characteristics. Processing can be usedto reject static objects like chairs and dynamic objects like fans.Thereafter, a two or three dimensional “map” of the location of thepeople in a room can be sent to AEC/Beamformer device 204/206 accordingto aspects of the embodiments in the form of a user location data signal(which would include the locations of one or more people using the areaor room in which system 200 is located), or area image data signal(which would include a two or three dimensional map of the area or room,and include locations of many different object such as furniture, fans(and not whether they were moving or not), and people, walls, floor inthe area or room in which system 200 is located, or the signals can becombined as one data signal output from WSS 250 and received byAEC/Beamformer device 204/206 according to aspects of the embodiments.

The performance of beamformer 206 in APS 200 can be optimized bycombining the output of WSS 250, as described above, with beamformer206. According to aspects of the embodiments, by measuring the distanceto the floor, table and walls, and providing such distance informationto beamformer 206, as shown in FIG. 2, the processor(s) in beamformer206 can then optimize beam parameters such as beam width, beam receptionangle, and ranges, to determine an optimum pickup pattern for a givenroom. According to aspects of the embodiments, there are numerous waysin which distance information can be used to shape beamformingscenarios. By way of a non-limiting example, the beam can be narrowed asthe distance increases to improve the signal to noise ratio (SNR).According to a further non-limiting example, the distance informationcan be used to adjust the gain of the audio beam (meaning adjusting thesensitivity of the speaker and/or the gain coefficients for thatparticular speaker's output in beam formation). According to furtheraspects of the embodiments, the distance information can be used to makea three dimensional (3D) map of the room and the twoangles-plus-distance can be converted to an XYZ map. According tofurther aspects of the embodiments, the 3D map and/or XYZ map can thenbe used to tell if something or someone were within the room boundaryand we noise can be ignored from outside the room. Beamformer 206 canthen adjust the beams to reduce the reflected noise coming off a wall.Furthermore, if beamformer 206 knows the location of every person in aroom, beamformer 206 can be programmed to not adapt the beam to receivevoices coming from outside the areas where people are located, or, usingthe data output form WSS 250, beamformer 206 can simply ignore audio, asmuch as possible, that originates outside the geographical area of theroom. This can prevent targeting the beam to an open doorway therebysubstantially reducing or eliminating hallway voice pickup, or to atransducer-speaker outputting the far end speech (as is shown in FIG. 2;speakers 208 a,b).

According to further aspects of the embodiments, by targeting beams toonly people 104 speaking in a room, the beamformer can reject orsubstantially reject all of the sound coming from a television orstereo, which thereby improves the use of intercoms or voice recognitionservices, like Alexa® from Amazon®. According to still further aspectsof the embodiments, a beamformer that uses WSS 125 does not necessarilyrequire a far end or A/V reference signal to cancel the detection ofthis “noise” in the AEC/beamforming system, although sending a referenceto mic array 203 and incorporating AEC 204 can improve the undesiredsignal rejection further.

According to further aspects of the embodiments, improvements tobeamforming performance are not solely limited to dynamic beamformingsystems. Use of WSS 250 can also improve the performance of the fixedbeam setup. As with the dynamic system, a distance map can be generatedand used to identify a table in a room. According to aspects of theembodiments, a grid of fixed beam positions covering the outsideperimeter of table 106 would be one advantageous starting point, amongothers, for setup of the fixed beam array. According to further aspectsof the embodiments, the positon information generated by WSS 250 canalso be used to make a fixed beam approach behave in a manner similar tothat of an adaptive fixed beam. With such position information,beamformer 206 can configure the beams to point at people 104 in theroom. As long as there are enough beam outputs available for people 104in the room, a targeted beam can be placed on each person 104. If thenumber of people 104 exceeds the number of beams then the beam widthsand shapes can be reconfigured to cover more than one person, accordingto further aspects of the embodiments.

Also shown in FIGS. 2 and 3 is processor 220, memory 222, and thefollowing software or applications (Apps): acoustic echo cancellationsoftware/application (AEC App) 224, adaptive beamformingsoftware/application (ABF App) 226, active noise reductionsoftware/application (ANR App) 228, and conference systemsoftware/application (CS App) 230, as well as network 232. Apps 224,226, 228, and 230 are stored in memory 224 associated with processors222. Processors 222 and network 232 are described in greater detailbelow in regard to FIGS. 7 and 8, respectively. As those of skill in theart can now appreciate, each of AECD 204, adaptive beamformer circuitry206, ANR circuit 208, and network interface (audio conference computer)210 comprise processing devices and associated software. According toaspects of the embodiments, devices 204, 206, 208, and 210, eachcomprise one or more processors 220 and Apps 224, 226, 228, and 230; inregard to App 230, this can be embodied as a single, larger App, or canbe implemented as separate modules or Apps 224, 226, and 228 in each of204, 206, 208, and 210, respectively (as is shown in FIGS. 2 and 3). InAECD 204 App 224 and processor 222 perform the acoustic echocancellation processing on the audio signals; in adaptive beamformer206, App 226 and processor 222 perform the beamforming process on thereceived digital audio signals; in ANR circuit 208, App 228 andprocessor 222 perform active noise reduction; and in audio conferencecomputer 210, processor 222 and App 230 perform the network interfaceprocessing. As those of skill in the art can further appreciate, theactive noise reduction, beamforming, acoustic echo cancellation andnetwork communications functions can all be performed by the separatedevices as shown as well as by a single device, with one or more Apps224, 226, 228, and 230. That is, substantially all of the processing canbe performed in a single processing device, such as a laptop computer,desktop computer, cell-phone, tablet, among other types of processingdevices (e.g., 204, 206, 208, and 210 can all be one device, such as alaptop, or cell phone, with App 230 performing the processing asdescribed herein).

While some embodiments will be described in the general context ofprogram modules that execute in conjunction with an application programthat runs on an operating system on a personal computer, or otherprocessing devices, those skilled in the art will recognize that aspectsmay also be implemented in combination with other program modules.

Generally, program modules include routines, programs, components, datastructures, and other types of structures that perform particular tasksor implement particular abstract data types. Moreover, those of skill inthe art can appreciate that different aspects of the embodiments can bepracticed with other computer system configurations, including hand-helddevices, multiprocessor systems, microprocessor-based or programmableconsumer electronics, minicomputers, mainframe computers, and comparablecomputing devices. Aspects of the embodiments can also be practiced indistributed computing environments where tasks are performed by remoteprocessing devices that are linked through a communications network. Ina distributed computing environment, program modules can be located inboth local and remote memory storage devices.

Aspects of the embodiments can be implemented as a computer-implementedprocess (method), a computing system, or as an article of manufacture,such as a computer program product or computer readable media. Thecomputer program product can be a computer storage medium readable by acomputer system and encoding a computer program that comprisesinstructions for causing a computer or computing system to performexample process(es). The computer-readable storage medium is acomputer-readable memory device. The computer-readable storage mediumcan for example be implemented via one or more of a volatile computermemory, a non-volatile memory, a hard drive, a flash drive, a floppydisk, or a compact disk, and comparable hardware media.

Throughout this specification, the term “platform” can be a combinationof software and hardware components for processing audio signals forbeamforming and acoustic echo cancellation according to aspects of theembodiments. Examples of platforms include, but are not limited to, ahosted service executed over a plurality of servers, an applicationexecuted on a single computing device, and comparable systems. The term“server” generally refers to a computing device executing one or moresoftware programs typically in a networked environment. More detail onthese technologies and example operations is provided below.

A computing device, as used herein, refers to a device comprising atleast a memory and one or more processors that includes a server, adesktop computer, a laptop computer, a tablet computer, a smart phone, avehicle mount computer, or a wearable computer. A memory can be aremovable or non-removable component of a computing device configured tostore one or more instructions to be executed by one or more processors.A processor can be a component of a computing device coupled to a memoryand configured to execute programs in conjunction with instructionsstored by the memory. Actions or operations described herein may beexecuted on a single processor, on multiple processors (in a singlemachine or distributed over multiple machines), or on one or more coresof a multi-core processor. An operating system is a system configured tomanage hardware and software components of a computing device thatprovides common services and applications. An integrated module is acomponent of an application or service that is integrated within theapplication or service such that the application or service isconfigured to execute the component. A computer-readable memory deviceis a physical computer-readable storage medium implemented via one ormore of a volatile computer memory, a non-volatile memory, a hard drive,a flash drive, a floppy disk, or a compact disk, and comparable hardwaremedia that includes instructions thereon to automatically save contentto a location. A user experience can be embodied as a visual displayassociated with an application or service through which a user interactswith the application or service. A user action refers to an interactionbetween a user and a user experience of an application or a userexperience provided by a service that includes one of touch input,gesture input, voice command, eye tracking, gyroscopic input, pen input,mouse input, and keyboards input. An application programming interface(API) can be a set of routines, protocols, and tools for an applicationor service that allow the application or service to interact orcommunicate with one or more other applications and services managed byseparate entities.

Aspects of the embodiments address a need that arises from very largescale of operations created by networked computing and cloud-basedservices that cannot be managed by humans. The actions/operationsdescribed herein are not a mere use of a computer, but address resultsof a system that is a direct consequence of software used as a servicesuch as communication services offered in conjunction withcommunications.

While some embodiments will be described in the general context ofprogram modules that execute in conjunction with an application programthat runs on an operating system on a personal computer, those skilledin the art will recognize that aspects may also be implemented incombination with other program modules.

Generally, program modules include routines, programs, components, datastructures, and other types of structures that perform particular tasksor implement particular abstract data types. Moreover, those skilled inthe art can appreciate that aspects of the embodiments can be practicedwith other computer system configurations, including hand-held devices,multiprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers, and comparablecomputing devices. Aspects of the embodiments can also be practiced indistributed computing environments where tasks are performed by remoteprocessing devices that are linked through a communications network. Ina distributed computing environment, program modules can be located inboth local and remote memory storage devices.

Some aspects of the embodiments can be implemented as acomputer-implemented process (method), a computing system, or as anarticle of manufacture, such as a computer program product or computerreadable media. The computer program product can be a computer storagemedium readable by a computer system and encoding a computer programthat comprises instructions for causing a computer or computing systemto perform example process(es). The computer-readable storage medium isa computer-readable memory device. The computer-readable storage mediumcan, for example, be implemented via one or more of a volatile computermemory, a non-volatile memory, a hard drive, a flash drive, a floppydisk, or a compact disk, and comparable hardware media, among othertypes of storage media.

A computing device, as used herein, refers to a device comprising atleast a memory and one or more processors that includes a server, adesktop computer, a laptop computer, a tablet computer, a smart phone, avehicle mount computer, or a wearable computer. A memory can be aremovable or non-removable component of a computing device adapted tostore one or more instructions to be executed by one or more processors.A processor can be a component of a computing device coupled to a memoryand adapted to execute programs in conjunction with instructions storedby the memory. Actions or operations described herein can be executed ona single processor, on multiple processors (in a single machine ordistributed over multiple machines), or on one or more cores of amulti-core processor. An operating system can be a system adapted tomanage hardware and software components of a computing device thatprovides common services and applications. An integrated module can be acomponent of an application or service that can be integrated within theapplication or service such that the application or service can beadapted to execute the component. A computer-readable memory device canbe a physical computer-readable storage medium implemented via one ormore of a volatile computer memory, a non-volatile memory, a hard drive,a flash drive, a floppy disk, or a compact disk, and comparable hardwaremedia that includes instructions thereon to substantially automaticallysave content to a location. A user experience can be a visual displayassociated with an application or service through which a user interactswith the application or service. A user action refers to an interactionbetween a user and a user experience of an application or a userexperience provided by a service that includes one of touch input,gesture input, voice command, eye tracking, gyroscopic input, pen input,mouse input, and keyboards input, among other types of inputs. An APIcan be a set of routines, protocols, and tools for an application orservice that allow the application or service to interact or communicatewith one or more other applications and services managed by separateentities.

Attention is directed to FIG. 3, which illustrates a block diagram ofAPS 300 using fixed beamforming, with beamforming before AEC, which canbe described as using an adaptive beamformer 206 with AEC beforebeamforming for use with room 100. APS 300 comprises beamformer 206,AECDs 204 a-m, ANR circuits 208 a-m, N:1 auto mixer 302, and WSS 250,among other devices, according to aspects of the embodiments. Accordingto aspects of the embodiments, use of WSS 250 generates positioninformation to re-configure the beams in a fixed beamforming system. Thegeneration and use of position information to reconfigure the beams doesnot necessarily require calculating direction of arrival or voiceactivity from the audio signals picked up by mic array 203. Thereconfiguration of the previously fixed beams can be referred to as“beam adaption”, which is a hybrid system that has multiple adaptingbeams. As those of skill in the art can appreciate, unlike traditionaladaptive beamforming, the systems of FIGS. 2 and 3, APSs 200, 300 do notuse direction of arrival or other audio information to adapt the beams,though according to further aspects of the embodiments, such functionscan be implemented in the circuits of FIGS. 2 and 3.

Use of an acoustic echo canceler, by way of non-limiting example AECD204, on each beam signal can also be used to remove far end signal fromthe mic signal; this is shown in FIG. 3, with reference line 218carrying the far end audio signal from Ethernet interface 210 to eachAECD 204 a-m. According to still further aspects of the embodiments, therate of beam adaption by beamformer 206 should be performed no fasterthan that which the echo canceller AECD 204 can adapt to; that is, theecho cancellation rate of AECD 204 and the beam formation rate ofbeamformer 206 are substantially equivalent. For example, if theadaption rate of the echo canceler is known, then the beam positioningshould be changed slow enough so that the echo cancel filter adaptioncan keep up. According to still further aspects of the embodiments,noise reduction can then be used to remove extraneous noise such as thatgenerated by air conditioning of heating systems (HVAC noise). Finally,according to still further aspects of the embodiments, an auto-mixer cancombine the output of the multitude of beams to produce a singlenear-end speech signal to send to the far end.

In order of operation, APS 300 generates a plurality of beams (usingdata from WSS 250 according to aspects of the embodiments), as describedabove, each of which is processed by a respective AECD 204 a-m, and thenactive noise reduction is performed by a respective ANR circuit 208 a-m.The outputs of each of ANR circuits 208 a-m are input to N:1 auto-mixer302, which generates only one output signal to send to the far end viaEthernet interface 210.

The result of implementing WSS 250 in the beamforming system results inperformance superior to that than a fully adaptive beamformer whenmultiple people 104 speak in room 100. A fully adaptive beamformer canproduce only one signal so it would typically favor one person 104speaking over another. This is not as natural sounding as the multi-beamapproach.

According to still further aspects of the embodiments, combining micarray 203 and WSS 250 into a single POE powered network wall or ceilingmounted device can eliminate the need for other devices that arenormally required in a modern home or office space. For example, WSS 250can map all the devices or objects in room 100, as well as people 104,and detect their motion. Separate motion detectors, which are often usedto control lights, adjust the room temperatures, and turn on/off A/Vequipment, can be eliminated. Such implementations do not have to bestand alone devices, but can be combined into one apparatus, as the rateof data sampling by WSS 250 and data processing rates of other devicesmakes such multiple uses relatively straightforward. In addition, WSS250 can be used to count the number of people 104 in room 100 that canthen be reported to a cloud-based management system, such as Crestron'sXiO Cloud, to track room utilization. By tracking the number of people104 in meetings spaces 100, companies can optimize future spaceplanning.

According to still further aspects of the embodiments, WSS 250 can beused to recognize hand and arm movement gestures due to its highresolution. In recognizing gestures, WSS 250 can be used to controlvarious aspects of the AV system, among other devices. By way ofnon-limiting example, raising an arm can be recognized as a request toincrease the volume of audio or increase the intensity of lights; or, asideways swipe of the arm and hand can be used to inform a computer toadvance a Power Point slide. According to still further aspects of theembodiments, gesture control can be limited to a person 104 seated atthe head of table 106, so that arbitrary hand motions by untrainedpeople sitting at conference table 106 do not cause unexpected behavior.Or, according to still further aspects of the embodiments, a specifichand or arm motion can first be performed in order for WSS 250 tointerpret subsequent gesture controls; in this manner, control of one ormore devices can be handed off from one person 104 a to the next person104 b, regardless of position in room 100.

According to still further aspects of the embodiments, additionalantennas 212 and/or multiple MWSs 250 can be added to increase thecapabilities of both APS 200, 300 (note that FIG. 4 illustrates animplementation of two antennas 212). The incremental cost is small sincemuch of the cost is in the microprocessor, audio DSP, power supplies andenclosure. According to further aspects of the embodiments, theimplementations of APSs 200, 300 can be embodied in as few as a coupleof different devices, or, as is shown in each of the Figures, differentdevices for different functionalities. According to still furtheraspects of the embodiments, additional sensors can be added such aslight, temperature, and humidity sensors, among others. Such anintegrated system can provide control over lighting (lights, shades,curtain), HVAC settings, as well as video and audio applications, byusing the determination of the presence and/or motion of the occupantsusing MWSs 125.

FIG. 5 illustrates a flow chart of method 500 for operating audioprocessing system 200 as shown in FIG. 2, in a conference room in whicha plurality of microphones 202 output audio signals to an equalplurality of acoustic echo cancellation devices 204 that provide aplurality of echo cancelled audio signals to an adaptive beamformingdevice 206 that receives as an input a room image signal (from WSS 250)to facilitate generation of an audio beam signal subject to active noisereduction in ANR circuit 208 prior to be sent to a far end conferenceroom by NW I/F 210 according to aspects of the embodiments. The roomimage signal generated by WSS 250 can include two or three dimensionalroom data, i.e., a layout of the room, or area within which audioprocessing system 300 is located, and/or information about locations ofone or more people (users) in the room or area within which audioprocessing system is located.

Attention is directed to FIG. 5 and method 500, which begins with methodstep 502. In method step 502 acoustic audio signals are received by mics202 a-n. In method step 504 mics 202 output electrical audio signals toan equal plurality of AECDs 204 a-n (method step 506). The outputelectrical audio signals can be analog signals, but typically aredigital, wherein mics 202 include analog-to-digital converters.

In method step 506 each AECD 204 performs acoustic echo cancellation onthe received mic audio signals using algorithms and processing that alsoincorporate reference signal 218 that has been received from a far endconference room (not shown) by NW I/F 210. In method step 508, theoutput of WSS 250 is received by adaptive beamforming circuit 206wherein a single audio beam is generated or adapted taking into accountthe information contained in the room image data signal generated by WSS250 (method step 510). In method step 512 the single audio beamgenerated or adapted by adaptive beam forming circuit 206 is transmittedto ANR circuit 208, and in method step 514 noise cancellation processingis performed on the audio beam signal output by adaptive beamformingcircuit 206. In method step 516, the noise reduced audio signal istransmitted by ANR circuit 208 to NW I/F 210, where it is thentransmitted to the far end conference room as a near end audio signal.

FIG. 6 illustrates a flow chart of method 600 for operating audioprocessing system 300 as shown in FIG. 3, in a conference room in whicha plurality of microphones 202 output audio signals to adaptivebeamforming device 206 that also receives as an input a room imagesignal from WSS 250 to facilitate generation of a plurality of audiobeam signals that are then processed individually by AEDs 204 and ANRcircuits 208 prior to being auto-mixed by auto mixer 302 and sent to afar end conference room by NW I/F 210 according to aspects of theembodiments. The room image signal generated by WSS 250 can include twoor three dimensional room data, i.e., a layout of the room, or areawithin which audio processing system 300 is located, and/or informationabout locations of one or more people (users) in the room or area withinwhich audio processing system is located.

Attention is directed to FIG. 6 and method 600, which begins with methodstep 602. In method step 602 acoustic audio signals are received by mics202 a-m. The output electrical audio signals can be analog signals, buttypically are digital, wherein mics 202 a-m include analog-to-digitalconverters. In method step 604 each of mics 202 a-m transmits itsrespective electrical mic output audio signal, which are then receivedby adaptive beamforming circuit 206 in method step 606.

In method step 608 WSS 250 generates the room image data signal andtransmits the same to adaptive beamforming circuit 206. In method step610 adaptive beamforming circuit 206 receives the room image data signaland generates or adapts a plurality of audio beam signals taking intoaccount the information contained in the room image data signalgenerated by WSS 250 and the received mic audio signals.

In method step 612, adaptive beamforming circuit 206 outputs theplurality of audio beam signals generated in method step 610 to an equalplurality of AECDs 204 a-m. Also, in method step 612, each AECD 204 a-malso receives the reference signal 218 from NW I/F 210, which is the farend conference room audio signal that is also output to one or morespeakers (not shown) in the conference room. Echo cancellation isperformed on each audio beam signal taking into account the referencesignal.

In method step 614, a plurality of acoustic echo cancelled signals areoutput from the respective plurality of AECDs 204 a-m and transmitted toa respective plurality of ANR circuits 208 a-m. Each ANR circuit 208 a-mthen performs noise reduction processing on the received acoustic echocancelled signals, and a plurality of noise reduced audio signals arethen output from respective ANR circuits 208 a-m.

In method step 616, the plurality of noise reduced audio signals arereceived by an N:1 auto mixer that combines the received signals andoutputs a single combined audio output signal. In method step 618, thesingle combined audio output signal is received by NW I/F 210 andtransmitted to the far end conference room.

FIG. 7 illustrates a personal computer/processor/laptop suitable for useto implement the methods shown in FIGS. 5 and 6, among other methods,for optimizing adaptive beamforming according to aspects of theembodiments.

FIG. 7 illustrates a block diagram of NW I/F or audio conferencecomputer 210 ((from hereon in, NW I/F 210) and other types of computers,such as laptops, desktops, tablets, personal digital assistants (PDAs)and the like) suitable for use to implement methods 500 and 600 forperforming adaptive beamforming according to aspects of the embodiments.NW I/F 210 comprises, among other items, shell/box 701, integrateddisplay/touch-screen (display) 702 (though not used in every applicationof NW I/F 210), internal data/command bus (bus) 704, processorboard/processor internal memory (internal memory) 732, and one or moreprocessors 124 with processor internal memory 706 (which can betypically read only memory (ROM) and/or random access memory (RAM)).Those of ordinary skill in the art can appreciate that in modernprocessor systems, parallel processing is becoming increasinglyprevalent, and whereas a single processor would have been used in thepast to implement many or at least several functions, it is more commoncurrently to have a single dedicated processor for certain functions(e.g., digital signal processors) and therefore could be severalprocessors, acting in serial and/or parallel, as required by thespecific application. NW I/F 210 further comprises multiple input/outputports, such as universal serial bus ports 710, Ethernet ports 711, andvideo graphics array (VGA) ports/high definition multimedia interface(HDMI) ports 722, among other types. Further, NW I/F 210 includesexternally accessible drives such as compact disk (CD)/digital videodisk (DVD) read/write (RW) (CD/DVD/RW) drive 712, and floppy diskettedrive 714 (though less used currently, many computers still include thisdevice). NW I/F 210 still further includes wireless communicationapparatus, such as one or more of the following: Wi-Fi transceiver 742,BlueTooth (BT) transceiver 744, near field communications (NFC)transceiver 746, third generation (3G)/fourth Generation (4G)/long termevolution (LTE) (3G/4G/LTE) transceiver 748, communicationssatellite/global positioning system (satellite) transceiver device 750,and antenna 752.

Internal memory 732 itself can comprise hard disk drive (HDD) 716 (thesecan include conventional magnetic storage media, but, as is becomingincreasingly more prevalent, can include flash drive memory 734, amongother types), read-only memory (ROM) 718 (these can include electricallyerasable (EE) programmable ROM (EEPROMs), ultra-violet erasable PROMs(UVPROMs), among other types), and random access memory (RAM) 720.Usable with USB port 710 is flash drive memory 734, and usable withCD/DVD/RW drive 712 are CD/DVD disks 736 (which can be both read andwrite-able). Usable with floppy diskette drive 714 are floppy diskettes738. External memory storage 724 can be used to store data and programsexternal to box 701 of audio conference computer 120, and can itselfcomprise another HDD 716 a, flash drive memory 734 (which can also bereferred to as “storage media”), among other types of memory storage.External memory storage 724 is connectable to NW I/F 210 via USB cable756. Each of the memory storage devices, or the memory storage media(706, 716, 718, 720, 724, 734, 736, and 738, among others), can containparts or components, or in its entirety, executable software programmingcode or application (application, or “App”) Apps 224, 226, 228, and 230,which can implement part or all of the portions of methods 500 and 600described herein. In FIG. 7, Apps 224, 226, 228, and 230 have beenrepresented by the designation “XXX.”

In addition to the above described components, NW I/F 210 also compriseskeyboard 728, external display 726, printer/scanner/fax machine 760, andmouse 730 (although not technically part of processor 124, theperipheral components as shown in FIGS. 7 (724, 726, 728, 730, 734, 736,738, and 760) are so well known and adapted for use with NW I/F 210 thatfor purposes of this discussion they shall be considered as being partof audio conference computer 120). Other cable types that can be usedwith NW I/F 210 include RS 232, among others, not shown, that can beused for one or more of the connections between NW I/F 210 and theperipheral components described herein. Keyboard 728, mouse 730, andprinter/scanner/fax machine 760 are connectable to NW I/F 210 via USBcable 756 and USB ports 710, and external display 726 is connectible tocomputer 120 via VGA cable/HDMI cable 723. NW I/F 210 is connectible tonetwork 122 (which can be the Internet) via Ethernet port 77 andEthernet cable 758 via a router and modulator-demodulator (MODEM),neither of which are shown in FIG. 7. All of the immediatelyaforementioned components (722, 724, 726, 728, 730, 734, 736, 738, 756,758, and 760) are known to those of ordinary skill in the art, and thisdescription includes all known and future variants of these types ofdevices.

External display 726 can be any type of known display or presentationscreen, such as liquid crystal displays (LCDs), light emitting diodedisplays (LEDs), plasma displays, cathode ray tubes (CRTs), amongothers. In addition to the user interface mechanism such as mouse 730,NW I/F 210 can further include a microphone, touch pad, joy stick, touchscreen, voice-recognition system, among other inter-activeinter-communicative devices/programs, which can be used to enter dataand voice, and which all of are known to those of skill in the art andthus a detailed discussion thereof has been omitted in fulfillment ofthe dual purposes of clarity and brevity.

As mentioned above, NW I/F 210 further comprises a plurality of wirelesstransceiver devices, such as Wi-Fi transceiver 742, BT transceiver 744,NFC transceiver 746, 3G/4G/LTE transceiver 748, satellite transceiverdevice 750, and antenna 752. While each of Wi-Fi transceiver 742, BTtransceiver 744, NFC transceiver 746, 3G/4G/LTE transceiver 748, andsatellite transceiver device 750 has their own specialized functions,each can also be used for other types of communications, such asaccessing a cellular service provider (not shown), accessing theInternet, texting, emailing, among other types communications anddata/voice transfers/exchanges, as known to those of skill in the art.Each of Wi-Fi transceiver 742, BT transceiver 744, NFC transceiver 746,3G/4G/LTE transceiver 748, satellite transceiver device 750 includes atransmitting and receiving device, and a specialized antenna, althoughin some instances, one antenna can be shared by one or more of Wi-Fitransceiver 742, BT transceiver 744, NFC transceiver 746, 3G/4G/LTEtransceiver 748, and satellite transceiver device 750. Alternatively,one or more of Wi-Fi transceiver 742, BT transceiver 744, NFCtransceiver 746, 3G/4G/LTE transceiver 748, and satellite transceiverdevice 750 will have a specialized antenna, such as satellitetransceiver device 750 to which is electrically connected at least oneantenna 752.

In addition, NW I/F 210 can access network 122, either through ahard-wired connection such as Ethernet port 77 as described above, orwirelessly via Wi-Fi transceiver 742, 3G/4G/LTE transceiver 748 and/orsatellite transceiver 750 (and their respective antennas) according toaspects of the embodiments. NW I/F 210 can also be part of a largernetwork configuration as in a global area network (GAN) (e.g., theinternet), which ultimately allows connection to various landlines.

According to further aspects of the embodiments, integrated touch screendisplay 702, keyboard 728, mouse 730, and external display 726 (if inthe form of a touch screen), can provide a means for a user to entercommands, data, digital, and analog information into audio conferencecomputer 120. Integrated and external displays 702, 726 can be used toshow visual representations of acquired data, and the status ofapplications that can be running, among other things.

Bus 704 provides a data/command pathway for items such as: the transferand storage of data/commands between audio conference computer 120,Wi-Fi transceiver 742, BT transceiver 744, NFC transceiver 746,3G/4G/LTE transceiver 748, satellite transceiver device 750, integrateddisplay 702, USB port 710, Ethernet port 77, VGA/HDMI port 722,CD/DVD/RW drive 712, floppy diskette drive 714, and internal memory 732.Through bus 704, data can be accessed that is stored in internal memory732. NW I/F 210 can send information for visual display to either orboth of integrated and external displays 702, 726, and the user can sendcommands to system operating programs/software/Apps (including Apps 224,226, 228, and 230) that might reside in processor internal memory 706 ofaudio conference computer 120, or any of the other memory devices (736,738, 716, 718, and 720).

NW I/F 210 and either processor internal memory 706 or internal memory732, can be used to implement methods 500 and 600, among others, forperforming adaptive beamforming according to aspects of the embodiments.Hardware, firmware, software or a combination thereof may be used toperform the various steps and operations described herein. According toaspects of the embodiments, one or more of Apps 224, 226, 228, and 230for carrying out the above discussed steps can be stored and distributedon multi-media storage devices such as devices 716, 718, 720, 734, 736and/or 738 (described above) or other form of media capable of portablystoring information. Storage media 734, 736 and/or 738 can be insertedinto, and read by devices such as USB port 710, CD/DVD/RW drive 712, anddisk drives 714, respectively.

As also will be appreciated by one skilled in the art, the variousfunctional aspects of the embodiments may be embodied in a wirelesscommunication device, a telecommunication network, or as one or moremethods (500, 600, among others) or in a computer program product.Accordingly, the embodiments may take the form of an entirely hardwareembodiment or an embodiment combining hardware and software aspects.Further, the embodiments may take the form of a computer program productstored on a computer-readable storage medium having computer-readableinstructions embodied in the medium. Any suitable computer-readablemedium may be utilized, including hard disks, CD-ROMs, digital versatilediscs (DVDs), optical storage devices, or magnetic storage devices sucha floppy disk or magnetic tape. Other non-limiting examples ofcomputer-readable media include flash-type memories or other known typesof memories.

Further, those of ordinary skill in the art in the field of theembodiments can appreciate that such functionality can be designed intovarious types of circuitry, including, but not limited to fieldprogrammable gate array structures (FPGAs), application specificintegrated circuitry (ASICs), microprocessor based systems, among othertypes. A detailed discussion of the various types of physical circuitimplementations does not substantively aid in an understanding of theembodiments, and as such has been omitted for the dual purposes ofbrevity and clarity. However, as well known to those of ordinary skillin the art, the systems and methods discussed herein can be implementedas discussed and can further include programmable devices.

Such programmable devices and/or other types of circuitry as previouslydiscussed can include a processing unit, a system memory, and a systembus that couples various system components including the system memoryto the processing unit. The system bus can be any of several types ofbus structures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures.Furthermore, various types of computer readable media can be used tostore programmable instructions. Computer readable media can be anyavailable media that can be accessed by the processing unit. By way ofexample, and not limitation, computer readable media can comprisecomputer storage media and communication media. Computer storage mediaincludes volatile and nonvolatile as well as removable and non-removablemedia implemented in any method or technology for storage of informationsuch as computer readable instructions, data structures, program modulesor other data. Computer storage media includes, but is not limited to,RAM, ROM, EEPROM, flash memory or other memory technology, CDROM,digital versatile disks (DVD) or other optical disk storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can be accessed by the processing unit.Communication media can embody computer readable instructions, datastructures, program modules or other data in a modulated data signalsuch as a carrier wave or other transport mechanism and can include anysuitable information delivery media.

The system memory can include computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) and/orrandom-access memory (RAM). A basic input/output system (BIOS),containing the basic routines that help to transfer information betweenelements connected to and between the processor, such as duringstart-up, can be stored in memory. The memory can also contain dataand/or program modules that are immediately accessible to and/orpresently being operated on by the processing unit. By way ofnon-limiting example, the memory can also include an operating system,application programs, other program modules, and program data.

The processor can also include other removable/non-removable andvolatile/nonvolatile computer storage media. For example, the processorcan access a hard disk drive that reads from or writes to non-removable,nonvolatile magnetic media, a magnetic disk drive that reads from orwrites to a removable, nonvolatile magnetic disk, and/or an optical diskdrive that reads from or writes to a removable, nonvolatile opticaldisk, such as a CD-ROM or other optical media. Otherremovable/non-removable, volatile/nonvolatile computer storage mediathat can be used in the operating environment include, but are notlimited to, magnetic tape cassettes, flash memory cards, digitalversatile disks, digital video tape, solid state RAM, solid state ROMand the like. A hard disk drive can be connected to the system busthrough a non-removable memory interface such as an interface, and amagnetic disk drive or optical disk drive can be connected to the systembus by a removable memory interface, such as an interface.

The embodiments discussed herein can also be embodied ascomputer-readable codes on a computer-readable medium. Thecomputer-readable medium can include a computer-readable recordingmedium and a computer-readable transmission medium. Thecomputer-readable recording medium is any data storage device that canstore data which can be thereafter read by a computer system. Examplesof the computer-readable recording medium include read-only memory(ROM), random-access memory (RAM), CD-ROMs and generally optical datastorage devices, magnetic tapes, flash drives, and floppy disks. Thecomputer-readable recording medium can also be distributed over networkcoupled computer systems so that the computer-readable code is storedand executed in a distributed fashion. The computer-readabletransmission medium can transmit carrier waves or signals (e.g., wiredor wireless data transmission through the Internet). Also, functionalprograms, codes, and code segments to, when implemented in suitableelectronic hardware, accomplish or support exercising certain elementsof the appended claims can be readily construed by programmers skilledin the art to which the embodiments pertains.

FIG. 8 illustrates network system 800 within which the systems andmethods shown in FIGS. 1-6 can operate for optimizing adaptivebeamforming according to aspects of the embodiments. Much of the networksystem infrastructure shown in FIG. 8 is or should be known to those ofskill in the art, so, in fulfillment of the dual purposes of clarity andbrevity, a detailed discussion thereof shall be omitted.

According to aspects of the embodiments, a user of the systems (200,300, among others) and methods (500, 600, among others) for performingadaptive beamforming can have Apps 224, 226, 228 and 230 on their mobiledevice (or cell phone) 802, as well as on NW I/F 210, laptop computer,server, tablet device, and/or dedicated devices 204, 206, 208 and 210,such as those shown in FIGS. 2 and 3, and the systems shown in theremaining Figures, according to aspects of the embodiments. Thus, eachof devices 802, 204, 206, 208, and 210 contain processor 220 and oneform or another of Apps 224, 226, 228, and 230 according to aspects ofthe embodiments. According to aspects of the embodiments, cell phone 802can include two or more mics 202 or two or more external mics 202 can beconnected via a wired or wired interface. According to further aspectsof the embodiments, two or more cell phones 802, each with a single mic202, or one cell phone 802 with two mics 202 (either internallyconnected or externally connected), with or without external speakers,can operate all of Apps 224, 226, 228 and 230 and perform one or moreboth of methods 500 and 600, among others, as discussed herein. The Apps224, 226, 228 and 230 can be integrated into one or more applicationsthat can be downloaded or previously stored onto cell phone 802 suchthat cell phone 802, or two or more of them, can operate as an audioconferencing system with one or more of adaptive beamforming, activenoise reductions, acoustic echo cancellation and the networkingcapabilities. According to further aspects of the embodiments, severalcell phones 802 a-n can operate the above described Apps 224, 226, 228and 230 so that the performance of the system is improved. According tostill further aspects of the embodiments, wave sensor system 250 can beconnected to the one or more cell phones 802 via a wired or wirelessconnection, as shown in FIG. 8.

Mobile devices 802 can include, but are not limited to, so-called smartphones, tablets, personal digital assistants, notebook and laptopcomputers, and essentially any device that can access the internetand/or cellular phone service or can facilitate transfer of the sametype of data in either a wired or wireless manner. For purposes of thisdiscussion, the user shall be discussed as using only mobile device 802,i.e., a smartphone, though such discussion should be understand to be ina non-limiting manner in view of the discussion above about the othertypes of devices that can access, use, and provide such information.

Mobile device 802 can access cellular service provider 814, eitherthrough a wireless connection (cellular tower 820) or via awireless/wired interconnection (a “Wi-Fi” system that comprises, e.g.,modulator/demodulator (modem) 808, wireless router 810, personalcomputer (PC) 804, internet service provider (ISP) 806, and network122). Further, mobile device 802 can include near field communication(NFC), “Wi-Fi,” and Bluetooth (BT) communications capabilities as well,all of which are known to those of skill in the art. To that end,network system 800 further includes, as many homes (and businesses) do,one or more PCs/servers 804 that can be connected to wireless router 810via a wired connection (e.g., modem 808) or via a wireless connection(e.g., Bluetooth). Modem 808 can be connected to ISP 806 to provideinternet-based communications in the appropriate format to end users(e.g., PC 804), and which takes signals from the end users and forwardsthem to ISP 806. Such communication pathways are well known andunderstand by those of skill in the art, and a further detaileddiscussion thereof is therefore unnecessary.

Mobile device 802 can also access global positioning system (GPS)satellite 828, which is controlled by GPS station 824, to obtainpositioning information (which can be useful for different aspects ofthe embodiments), or mobile device 802 can obtain positioninginformation via cellular service provider 814 using cell tower(s) 820according to one or more well-known methods of position determination.Some mobile devices 802 can also access communication satellites 818 andtheir respective satellite communication systems control stations 826(the satellite in FIG. 8 is shown common to both communications and GPSfunctions) for near-universal communications capabilities, albeit at amuch higher cost than convention “terrestrial” cellular services. Mobiledevice 802 can also obtain positioning information when near or internalto a building (or arena/stadium) through the use of one or more ofNFC/BT devices, the details of which are known to those of skill in theart. FIG. 8 also illustrates other components of network system 800 suchas plain old telephone service (POTS) provider 812 (though shown to beconnected to network 122 (which can be the Internet), connections havebeen omitted for clarity to devices 120 and 804).

According to further aspects of the embodiments, network system 800 alsocontains NW I/F 210, wherein one or more processors 220, using known andunderstood technology, such as memory, data and instruction buses, andother electronic devices, can store and implement code that canimplement the systems (200, 300, among others) and methods (500, 600among others) for performing adaptive beamforming according to aspectsof the embodiments.

The disclosed embodiments provide several different systems, softwareproducts, and methods generally related to audio systems and digitalsignal processing, and more particularly to systems, methods, and modesfor implementing a millimeter wave sensor to optimize operation of abeamforming microphone array, among other types of systems. It should beunderstood that this description is not intended to limit theembodiments. On the contrary, the embodiments are intended to coveralternatives, modifications, and equivalents, which are included in thespirit and scope of the embodiments as defined by the appended claims.Further, in the detailed description of the embodiments, numerousspecific details are set forth to provide a comprehensive understandingof the claimed embodiments. However, one skilled in the art wouldunderstand that various embodiments may be practiced without suchspecific details.

Although the features and elements of aspects of the embodiments aredescribed being in particular combinations, each feature or element canbe used alone, without the other features and elements of theembodiments, or in various combinations with or without other featuresand elements disclosed herein.

This written description uses examples of the subject matter disclosedto enable any person skilled in the art to practice the same, includingmaking and using any devices or systems and performing any incorporatedmethods. The patentable scope of the subject matter is defined by theclaims, and may include other examples that occur to those skilled inthe art. Such other examples are intended to be within the scope of theclaims.

The above-described embodiments are intended to be illustrative in allrespects, rather than restrictive, of the embodiments. Thus, theembodiments are capable of many variations in detailed implementationthat can be derived from the description contained herein by a personskilled in the art. No element, act, or instruction used in thedescription of the present application should be construed as criticalor essential to the embodiments unless explicitly described as such.Also, as used herein, the article “a” is intended to include one or moreitems.

All United States patents and applications, foreign patents, andpublications discussed above are hereby incorporated herein by referencein their entireties.

INDUSTRIAL APPLICABILITY

To solve the aforementioned problems, the aspects of the embodiments aredirected towards systems, methods, and modes for audio systems, and morespecifically to systems, methods, and modes for implementing amillimeter wave sensor to optimize operation of a beamforming microphonearray, as well as other home or enterprise systems.

Alternate Embodiments

Alternate embodiments may be devised without departing from the spiritor the scope of the different aspects of the embodiments.

The invention claimed is:
 1. A method for operating a beamformingmicrophone array for use in a predetermined area, the method comprising:receiving acoustic audio signals at each of a plurality of microphones,converting the same to an electrical mic audio signal, and outputtingeach of the plurality of electrical mic audio signals; generating a userlocation data signal by a wave sensor system, and outputting the userlocation data signal, wherein the user location data signal includeslocation information of one or more people within the predeterminedarea; receiving both the user location data signal and plurality ofecho-corrected mic audio signals at an adaptive beamforming device;adapting one or more beams by the adaptive beamforming device based onthe user location data signal and plurality of mic audio signals whereineach of the one or more beams acquires sound from one or more specificlocations in the predetermined area, wherein the predetermined area is aconference room, and if the user location data signal indicates thatthere are more people than beams that can be formed, then modifying, bythe adaptive beamforming circuit, one or more of the fixed beampositions to cover two or more people in the conference room such thateach person is covered by at least one fixed beam; and adjusting, by theadaptive beamforming circuit, a beam width and shape to cover two ormore people in the conference room.
 2. The method according to claim 1,wherein the wave sensor system comprises: a millimeter (mm) wavetransmitter; and a wave receiver.
 3. The method according to claim 1,wherein the wave sensor system comprises: an optical transmitter; and anoptical receiver.
 4. The method according to claim 1 further comprising:generating a three dimensional image of the predetermined area by thewave sensor system; and outputting the same as an area image datasignal.
 5. The method according to claim 4 further comprising: receivingthe area image data signal and the plurality of mic audio signals by theadaptive beamforming circuit; performing adaptive beamforming on theplurality of mic audio signals that takes into account the received areaimage data signal and the plurality of mic audio signals; and adaptingone or more beams to acquire sound from one or more specific locationsin the predetermined area.
 6. The method according to claim 5 furthercomprising: modifying the beam audio signals by the adaptive beamformingcircuit to reduce noise reflected off one or more objects within thepredetermined area based on the area image data signal.
 7. The methodaccording to claim 4, wherein the area image data signal comprises:information as to where motion is occurring within the predeterminedarea.
 8. The method according to claim 7, wherein the informationcontained within the area image data signal that motion is occurringwithin the predetermined area substantially eliminates objects from thearea image data signal that are substantially at rest.
 9. The methodaccording to claim 7, wherein the information contained within the areaimage data signal that motion is occurring within the predetermined areadoes not include objects that move with a substantially constantvelocity.
 10. The method according to claim 9, wherein the object thatmoves with a substantially constant periodicity comprises a fan.
 11. Themethod according to claim 4, wherein the area image data signalcomprises: distance information between the wave sensor system andobjects within the predetermined area.
 12. The method according to claim11, wherein the objects comprise one or more of a floor, table, walls,and other furniture.
 13. The method according to claim 11 furthercomprising: adapting one or more beams by the adaptive beamformingcircuit that takes into account the distance information generated bythe wave sensor system.
 14. The method according to claim 13 furthercomprising: modifying, by the adaptive beamforming circuit, one or moreof a beam width, beam reception angle, and range of the beam based onthe received distance information generated by the wave sensor system.15. The method according to claim 4, further comprising: receiving, bythe adaptive beamforming circuit, the area image data signal, the userlocation data signal, and the plurality of mic audio signals; andperforming adaptive beamforming on the plurality of mic audio signalsthat takes into account the information in the area image data signaland the user location data signal, such that the adaptive beamformingcircuit substantially ignores voice signals that originate from outsidethe areas where the users are located.
 16. The method according to claim4 further comprising: receiving, by the adaptive beamforming circuit,the area image data signal, the user location data signal, and theplurality of mic audio signals; and performing adaptive beamforming onthe plurality of mic audio signals that takes into account theinformation in the area image data signal and the user location datasignal, such that the adaptive beamforming circuit substantially ignoreaudio signals generated from one or more of a television and stereo. 17.The method according to claim 4, wherein the predetermined area is aconference room, there is at least one table located in the conferenceroom, and further wherein the area image data signal includesinformation as to a location of the at least one table in the conferenceroom, and the method further comprises: generating, by the adaptivebeamforming circuit, one or more fixed beam positions covering aperimeter of the at least one table in the conference room.
 18. Themethod according to claim 4 further comprising: determining, by anacoustic audio direction of arrival algorithm operating within theadaptive beamforming circuit, a direction of arrival of one or moremicrophone generated audio signals.
 19. The method according to claim 18further comprising: determining, by the direction of arrival algorithm,a direction of arrival of the one or more microphone generated audiosignals using information in the area image data signal received fromthe wave sensor system.
 20. The method according to claim 4 furthercomprising: determining, by the wave sensor system, motion of one ormore objects located in the predetermined area.
 21. The method accordingto claim 20, wherein the wave sensor system can include the objectmotion information about the predetermined area in the area image datasignal, and wherein the adaptive beamforming circuit can eliminate fixedobjects and objects moving at a substantially constant rate from thearea image data signal to determine a number of people located in thepredetermined area, and output the same as a room occupancy status. 22.The method according to claim 21, wherein the room occupancy status canbe used by other interconnected systems to control one or more oflights, temperature, and audio-video equipment in the conference room.23. The method according to claim 21, wherein the room occupancy statuscan be transmitted to a room monitoring system.
 24. The method accordingto claim 1, wherein the predetermined area comprises: a conference room.25. The method according to claim 1, further comprising: generating, bythe adaptive beamforming circuit, one or more beams to acquire soundfrom one or more specific locations in the predetermined area.
 26. Themethod according to claim 1, further comprising: receiving, by aplurality of acoustic echo cancellation devices (204), one for each ofthe plurality of microphones, the mic audio signal from a respective oneof the plurality of microphones; performing acoustic echo cancellationon the received mic audio signal; and outputting an echo-corrected micaudio signal.
 27. The method according to claim 26, further comprising:receiving, by a first communication device adapted to receive areference signal from a remote source, and forward the same to each ofthe one or more acoustic echo cancellation devices, and wherein each ofthe one or more acoustic echo cancellation devices is adapted to deletethe reference signal from a respective one of the microphone audiosignals received by the respective acoustic echo cancellation devices.28. The method according to claim 27, wherein the reference signalcomprises a far end audio signal.
 29. The method according to claim 1,wherein the wave sensor system is adapted to resolve distances withinthe predetermined area within about 1 mm and within about 1 degree. 30.The method according to claim 1, wherein the predetermined area is aconference room, and the method further comprises: extracting, by theadaptive beamforming circuit, location information for each person inthe conference room, and adapting a respective fixed beam position foreach person in the conference room.
 31. The method according to claim 1,wherein the adaptive beamforming circuit comprises: an automixeralgorithm, and wherein the method further comprises adapting, by theadaptive beamforming circuit, multiple beams and combing the multiplebeams to produce a single audio signal using the automixer algorithm.32. The method according to claim 1, further comprising: removing, by anactive noise reduction circuit, noise from an output of the adaptivebeamforming circuit, and outputting a noise reduced audio signal;receiving, by an Ethernet communication device, a far end audio signalfrom a remote location and outputting the same to one or more speakersand to each of the acoustic echo cancellation devices; receiving, by theEthernet communication device, as an input the noise reduced audiosignal from the active noise reduction circuit, and outputting the sameto the remote location; and extracting, by a power-over-Ethernet device,electrical power over Ethernet communications cables and providing theelectrical power to the circuits in the beamforming array.
 33. Themethod according to claim 32, wherein the predetermined area furthercomprises: one or more of each of light sensors, temperature sensors,and humidity sensors, and wherein the method further comprisesreceiving, by the beamforming microphone array, as inputs the outputsfrom each of the sensors, and outputting the sensor outputs through theEthernet communication device.
 34. The method according to claim 1,further comprising: recognizing, by the wave sensor system, gesturesincluding one or more of hand motion and arm motion.
 35. The methodaccording to claim 34, wherein the recognized gestures can control oneor more functions in the conference room, and wherein the functionsinclude one or more of lighting levels, audio levels, temperaturelevels, humidity levels, and positions of shades and/or curtains.
 36. Amethod for operating a beamforming microphone array for use in apredetermined area, the method comprising: receiving acoustic audiosignals at each of a plurality of microphones, converting the same to anelectrical mic audio signal, and outputting each of the plurality ofelectrical mic audio signals; generating a user location data signal bya wave sensor system, and outputting the user location data signal,wherein the user location data signal includes location information ofone or more people within the predetermined area; receiving both theuser location data signal and plurality of echo-corrected mic audiosignals at an adaptive beamforming device; adapting one or more beams bythe adaptive beamforming device based on the user location data signaland plurality of mic audio signals wherein each of the one or more beamsacquires sound from one or more specific locations in the predeterminedarea, wherein if the user location data signal indicates that there aremore people than beams that can be formed, then modifying, by theadaptive beamforming circuit, one or more of the fixed beam positions tocover two or more people in the conference room such that each person iscovered by at least one fixed beam; and adjusting, by the adaptivebeamforming circuit, a beam width and shape to cover two or more peoplein the conference room.
 37. A method for operating a beamformingmicrophone array for use in a predetermined area, the method comprising:receiving acoustic audio signals at each of a plurality of microphones,converting the same to an electrical mic audio signal, and outputtingeach of the plurality of electrical mic audio signals; generating a userlocation data signal by a wave sensor system, and outputting the userlocation data signal, wherein the user location data signal includeslocation information of one or more people within the predeterminedarea; receiving both the user location data signal and plurality ofecho-corrected mic audio signals at an adaptive beamforming device;adapting one or more beams by the adaptive beamforming device based onthe user location data signal and plurality of mic audio signals whereineach of the one or more beams acquires sound from one or more specificlocations in the predetermined area; removing, by an active noisereduction circuit, noise from an output of the adaptive beamformingcircuit, and outputting a noise reduced audio signal; and receiving, byan Ethernet communication device, a far end audio signal from a remotelocation and outputting the same to one or more speakers and to each ofthe acoustic echo cancellation devices.
 38. The method according toclaim 37, further comprising: receiving, by the Ethernet communicationdevice, as an input the noise reduced audio signal from the active noisereduction circuit, and outputting the same to the remote location; andextracting, by a power-over-Ethernet device, electrical power overEthernet communications cables and providing the electrical power to thecircuits in the beamforming array.